1. 5

    I think clarity of communication is one of the most underrated skills as a developer. If the ratio of reading code to writing code is 10:1 then the ratio of talking about code to writing code must be 100:1

    So, barring overtime, a truly senior developer must average no more than 24 minutes a week coding? Possible, for some people who have the title “architect”, but I doubt it’s true of that many developers. I hope that if I ever reach that point, it’s because my title is manager/consultant/whatever else.

    1.  

      I interpret these ratios as applying to time over the life of code, possibly including many people.

      Also, I am not aware of empirical studies that justify the stated ratios. I interpret them mostly as placeholders used to make a (useful) point.

      1.  

        I definitely know multiple people for whom this is true.

        I hope to avoid such a fate myself but it absolutely happens.

        1.  

          The majority of my time is doing operational stuff nowadays, and I frequently read code but am not an “official” developer on any of the projects whose code I’m reading. The ratio of loc read / loc written is > 10 for me for many projects.

        1. 3

          What does Rachel dislike about python?

          1. 8
            1. 0

              Oh wow: we are still learning lessons from the 90s aren’t we?

          1. 1

            I think I’m missing part of the picture. I get how using intrusive lists is good for performance, but is there something that prevented them from just copy-pasting the linked list code into every class?

            Of course, you wouldn’t want to do that. But the reason you wouldn’t want to do that seems like a standard story of what inheritance buys you: code reuse.

            1. 1

              In modern C++, I can use std::list<T> to make a linked list of objects of type T. I get the same efficient memory layout as with intrusive lists. But I don’t have to modify the type T to inherit from a list_node type. std::list works with any type, because it uses composition, not inheritance.

              In early C++, before templates were added, the only way to get this kind of performance was using inheritance and intrusive lists. Using inheritance for this purpose degrades code reuse. Composition is better.

            1. 5

              Without doing a bit of benchmarking, I’d assume that EnumSet will be slower than a bitfield. I would pick a bitfield in a case I thought it might matter.

              However, the benchmark in this post is bad. Let’s start with the small stuff: the benchmarks use straight-line loops, the code all three benchmarks in sequence in the same JVM[0], and the results are presented without units.

              The kicker is that the author notices that the JVM has optimized away one of his benchmark cases, and argues that this is fine, because it means that EnumSet “inhibits optimizations.” That’s not valid. It means that they inhibit dead code elimination in this particular case. Your actual code using bitfields will typically not be dead, so why measure that?

              [0] I can’t say whether the same JVM is a problem in this particular case, but it’s a bad practice in general.

              1. 50

                The paper has this to say (page 9):

                Regarding potential human research concerns. This experiment studies issues with the patching process instead of individual behaviors, and we do not collect any personal information. We send the emails to the Linux community and seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.

                [..]

                Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.

                I’m not familiar with the generally accepted standards on these kind of things, but this sounds rather iffy to me. I’m very far removed from academia, but I’ve participated in a few studies over the years, which were always just questionaries or interviews, and even for those I had to sign a consent waiver. “It’s not human research because we don’t collect personal information” seems a bit strange.

                Especially since the wording “we will have to report this, AGAIN, to your university” implies that this isn’t the first time this has happened, and that the kernel folks have explicitly objected to being subject to this research before this patch.

                And trying to pass off these patches as being done in good faith with words like “slander” is an even worse look.

                1. 78

                  They are experimenting on humans, involving these people in their research without notice or consent. As someone who is familiar with the generally accepted standards on these kinds of things, it’s pretty clear-cut abuse.

                  1. 18

                    I would agree. Consent is absolutely essential but just one of many ethical concerns when doing research. I’ve seen simple usability studies be rejected due to lesser issues.

                    It’s pretty clear this is abuse.. the kernel team and maintainers feel strongly enough to ban the whole institution.

                    1. 10

                      Yeah, agreed. My guess is they misrepresented the research to the IRB.

                      1. 3

                        They are experimenting on humans

                        This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                        1. 19

                          A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                          1. 0

                            A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                            This statement is so vague as to be almost meaningless. In what relevant ways is a professional penetration testing contract (or, more relevantly, the associated process) different from this particular research project? Which of the two should be held to a higher standard? Why? What does “held to a higher standard” even mean?

                            Moreover, that claim doesn’t actually have anything to do with the comment I was replying to, which was claiming that this project was “experimenting on humans”. It doesn’t matter whether or not something is “research” or “industry” for the purposes of whether or not it’s “human experimentation” - either it is, or it isn’t.

                            1. 18

                              Resident pentester and ex-academia sysadmin checking in. I totally agree with @Foxboron and their statement is not vague nor meaningless. Generally in a penetration test I am following basic NIST 800-115 guidance for scoping and target selection and then supplement contractual expectations for my clients. I can absolutely tell you that the methodologies that are used by academia should be held to a higher standard in pretty much every regard I could possibly come up with. A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data.

                              Let’s put it in real terms, I am hired to do a security assessment in a very fixed highly focused set of targets explicitly defined in contract by my client in an extremely fixed time line (often very short… like 2 weeks maximum and 5 day average). Guess what happens if social engineering is not in my contract? I don’t do it.

                              1. 1

                                Resident pentester and ex-academia sysadmin checking in.

                                Note: this is worded like an appeal to authority, although you probably don’t mean it that way, so I’m not going to act like you are.

                                I totally agree with @Foxboron and their statement is not vague nor meaningless.

                                Those are two completely separate things, and neither is implied by the other.

                                their statement is not vague nor meaningless.

                                Not true - their statement contained none of the information you just provided, nor any other sort of concrete or actionable information - the statement “hold to a higher standard” is both vague and meaningless by itself…and it was by itself in that comment (or, obviously, there were other words - none of them relevant) - there was no other information.

                                the methodologies that are used by academia should be held to a higher standard

                                Now you’re mixing definitions of “higher standard” - GP and I were talking about human experimentation and ethics, while you seem to be discussing rigorousness and reproducibility of experiments (although it’s not clear, because “A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data” is slightly ambiguous).

                                None of the above is relevant to the question of “was this a human experiment” and the closely-related one “is penetration testing a human experiment”. Evidence suggests “no” given that the term does not appear in that document, nor have I heard of any pentest being reviewed by an ethics review board, nor have I heard any mention of “human experimenting” in the security community (including when gray-hat and black-hat hackers and associated social engineering e.g. Kevin Mitnick are mentioned), nor are other similar, closer-to-human experimentation (e.g. A/B testing, which is far closer to actually experimenting on people) processes considered to be such - up until this specific case.

                              2. 5

                                if you’re an employee in an industry, you’re either informed of penetration testing activity, or you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment.

                                if a company did this to their employees without any warning, they’d be shitty too, but the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific is nonzero.

                                either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                1. 1

                                  you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment

                                  By this logic, you’ve also agreed to everything else in a massive, hundred-page long EULA that you click “I agree” on, as well as consent to be tracked by continuing to use a site that says that in a banner at the bottom, as well as consent to Google/companies using your data for whatever they want and/or selling it to whoever will buy.

                                  …and that’s ignoring whether or not companies that have pentesting done on them actually explicitly include that specific warning in your contract - “implicit” is not good enough, as then anyone can claim that, as a Linux kernel patch reviewer, you’re “implicitly agreeing that you may be exposed to the risk of social engineering for the purpose of getting bad code into the kernel”.

                                  the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific

                                  Like others, you’re mixing up the issue of whether the experiment was properly-designed with the issue of whether it was human experimentation. I’m not making any attempt to argue the former (because I know very little about how to do good science aside from “double-blind experiments yes, p-hacking no”), so I don’t know why you’re arguing against it in a reply to me.

                                  either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                  I completely agree that the goals are different - but again, that’s irrelevant for determining whether or not something is “human experimentation”. Doesn’t matter what the motive is, experimenting on humans is experimenting on humans.

                            2. 18

                              This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                              I had a former colleague that once bragged about getting someone fired at his previous job during a pentesting exercise. He basically walked over to this frustrated employee at a bar, bribed him a ton of money and gave a job offer in return for plugging a usb key into the network. He then reported it to senior management and the employee was fired. While that is an effective demonstration of a vulnerability in their organization, what he did was unethical under many moral frameworks.

                              1. 2

                                First, the researchers didn’t engage in any behavior remotely like this.

                                Second, while indeed an example of pentesting, most pentesting is not like this.

                                Third, the fact that it was “unethical under many moral frameworks” is irrelevant to what I’m arguing, which is that the study was not “human experimentation”. You can steal money from someone, which is also “unethical under many moral frameworks”, and yet still not be doing “human experimentation”.

                              2. 3

                                If there is a pentest contract, then there is consent, because consent is one of the pillars of contract law.

                                1. 1

                                  That’s not an argument that pentesting is human experimentation in the first place.

                            3. 42

                              The statement from the UMinn IRB is in line with what I heard from the IRB at the University of Chicago after they experimented on me, who said:

                              I asked about their use of any interactions, or use of information about any individuals, and they indicated that they have not and do not use any of the data from such reporting exchanges other than tallying (just reports in aggregate of total right vs. number wrong for any answers received through the public reporting–they said that much of the time there is no response as it is a public reporting system with no expectation of response) as they are not interested in studying responses, they just want to see if their tool works and then also provide feedback that they hope is helpful to developers. We also discussed that they have some future studies planned to specifically study individuals themselves, rather than the factual workings of a tool, that have or will have formal review.

                              They because claim they’re studying the tool, it’s OK to secretly experiment on random strangers without disclosure. Somehow I doubt they test new drugs by secretly dosing people and observing their reactions, but UChicago’s IRB was 100% OK with doing so to programmers. I don’t think these IRBs literally consider programmers sub-human, but it would be very inconvenient to accept that experimenting on strangers is inappropriate, so they only want to do so in places they’ve been forced to by historical abuse. I’d guess this will continue for years until some random person is very seriously harmed by being experimented on (loss of job/schooling, pushing someone unstable into self-harm, targeting someone famous outside of programming) and then over the next decade IRBs will start taking it seriously.

                              One other approach that occurs to me is that the experimenters and IRBs claim they’re not experimenting on their subjects. That’s obviously bullshit because the point of the experiment is to see how the people respond to the treatment, but if we accept the lie it leaves an open question: what is the role played by the unwitting subject? Our responses are tallied, quoted, and otherwise incorporated into the results in the papers. I’m not especially familiar with academic publishing norms, but perhaps this makes us unacknowledged co-authors. So maybe another route to stopping experimentation like this would be things like claiming copyright over the papers, asking journals for the papers to be retracted until we’re credited, or asking the universities to open academic misconduct investigations over the theft of our work. I really don’t have the spare attention for this, but if other subjects wanted to start the ball rolling I’d be happy to sign on.

                              1. 23

                                I can kind of see where they’re coming from. If I want to research if car mechanics can reliably detect some fault, then sending a prepared car to 50 garages is probably okay, or at least a lot less iffy. This kind of (informal) research is actually fairly commonly by consumer advocacy groups and the like. The difference is that the car mechanics will get paid for their work where as the Linux devs and you didn’t.

                                I’m gonna guess the IRBs probably aren’t too familiar with the dynamics here, although the researchers definitely were and should have known better.

                                1. 18

                                  Here it’s more like keying someone’s car to see how quick it takes them to get an insurance claim.

                                  1. 4

                                    Am I misreading? I thought the MR was a patch designed to fix a potential problem, and the issue was

                                    1. pushcx thought it wasn’t a good fix (making it a waste of time)
                                    2. they didn’t disclose that it was an auto-generated PR.

                                    Those are legitimate complaints, c.f. https://blog.regehr.org/archives/2037, but from the analogies employed (drugs, dehumanization, car-keying), I have to double-check that I haven’t missed an aspect of the interaction that makes it worse than it seemed to me.

                                    1. 2

                                      We were talking about Linux devs/maintainers too, I commented on that part.

                                      1. 1

                                        Gotcha. I missed that “here” was meant to refer to the Linux case, not the Lobsters case from the thread.

                                  2. 1

                                    Though there they are paying the mechanic.

                                  3. 18

                                    IRB is a regulatory board that is there to make sure that researchers follow the (Common Rule)[https://www.hhs.gov/ohrp/regulations-and-policy/regulations/common-rule/index.html].

                                    In general, any work that receives federal funding needs to comply with the federal guidelines for human subject research. All work involving human subjects (usually defined as research activities that involve interaction with humans) need to be reviewed and approved by the institution IRB. These approvals fall within a continuum, from a full IRB review (which involve the researcher going to a committee and explaining their work and usually includes continued annual reviews) to a declaration of the work being exempt from IRB supervision (usually this happens when the work meets one of the 7 exemptions listed in the federal guidelines). The whole process is a little bit more involved, see for example (all the charts)[https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts/index.html] to figure this out.

                                    These rules do not cover research that doesn’t involve humans, such as research on technology tools. I think that there is currently a grey area where a researcher can claim that they are studying a tool and not the people interacting with the tool. It’s a lame excuse that probably goes around the spirit of the regulations and is probably unethical from a research stand point. The data aggregation method or the data anonymization is usually a requirement for an exempt status and not a non-human research status.

                                    The response that you received from IRB is not surprising, as they probably shouldn’t have approved the study as non-human research but now they are just protecting the institution from further harm rather than protecting you as a human subject in the research (which, by the way, is not their goal at this point).

                                    One thing that sticks out to me about your experience is that you weren’t asked to give consent to participate in the research. That usually requires a full IRB review as informed consent is a requirement for (most) human subject research. Exempt research still needs informed consent unless it’s secondary data analysis of existing data (which your specific example doesn’t seem to be).

                                    One way to quickly fix it is to contact the grant officer that oversees the federal program that is funding the research. A nice email stating that you were coerced to participate in the research study by simply doing your work (i.e., review a patch submitted to a project that you lead) without being given the opportunity to provide prospective consent and without receiving compensation for your participation and that the research team/university is refusing to remove your data even after you contacted them because they claim that the research doesn’t involve human subjects can go a long way to force change and hit the researchers/university where they care the most.

                                    1. 7

                                      Thanks for explaining more of the context and norms, I appreciate the introduction. Do you know how to find the grant officer or funding program?

                                      1. 7

                                        It depends on how “stalky” you want to be.

                                        If NSF was the funder, they have a public search here: https://nsf.gov/awardsearch/

                                        Most PIs also add a line about grants received to their CVs. You should be able to match the grant title to the research project.

                                        If they have published a paper from that work, it should probably include an award number.

                                        Once you have the award number, you can search the funder website for it and you should find a page with the funding information that includes the program officer/manager contact information.

                                        1. 3

                                          If they published a paper about it they likely included the grant ID number in the acknowledgements.

                                          1. 1

                                            You might have more luck reaching out to the sponsored programs office at their university, as opposed to first trying to contact an NSF program officer.

                                        2. 4

                                          How about something like a an Computer Science - External Review Board? Open source projects could sign up, and include a disclaimer that their project and community ban all research that hasn’t been approved. The approval process could be as simple as a GitHub issue the researcher has to open, and anyone in the community could review it.

                                          It wouldn’t stop the really bad actors, but any IRB would have to explain why they allowed an experiment on subjects that explicitly refused consent.

                                          [Edit] I felt sufficiently motivated, so I made a quick repo for the project . Suggestions welcome.

                                          1. 7

                                            I’m in favor of building our own review boards. It seems like an important step in our profession taking its reponsibility seriously.

                                            The single most important thing I’d say is, be sure to get the scope of the review right. I’ve looked into this before and one of the more important limitations on IRBs is that they aren’t allowed to consider the societal consequences of the research succeeding. They’re only allowed to consider harm to experimental subjects. My best guess is that it’s like that because that’s where activists in the 20th-century peace movement ran out of steam, but it’s a wild guess.

                                            1. 4

                                              At least in security, there are a lot of different Hacker Codes of Ethics floating around, which pen testers are generally expected to adhere to… I don’t think any of them cover this specific scenario though.

                                              1. 2

                                                any so-called “hacker code of ethics” in use by any for-profit entity places protection of that entity first and foremost before any other ethical consideration (including human rights) and would likely not apply in a research scenario.

                                          2. 23

                                            They are bending the rules for non human research. One of the exceptions for non-human research is research on organization, which my IRB defines as “Information gathering about organizations, including information about operations, budgets, etc. from organizational spokespersons or data sources. Does not include identifiable private information about individual members, employees, or staff of the organization.” Within this exception, you can talk with people about how the organization merges patches but not how they personally do that (for example). All the questions need to be about the organization and not the individual as part of the organization.

                                            On the other hand, research involving human subjects is defined as any research activity that involves an “individual who is or becomes a participant in research, either:

                                            • As a recipient of a test article (drug, biologic, or device); or
                                            • As a control.”

                                            So, this is how I interpret what they did.

                                            The researchers submitted an IRB approval saying that they just downloaded the kernel maintainer mailing lists and analyzed the review process. This doesn’t meet the requirements for IRB supervision because it’s either (1) secondary data analysis using publicly available data and (2) research on organizational practices of the OSS community after all identifiable information is removed.

                                            Once they started emailing the list with bogus patches (as the maintainers allege), the research involved human subjects as these people received a test article (in the form of an email) and the researchers interacted with them during the review process. The maintainers processing the patch did not do so to provide information about their organization’s processes and did so in their own personal capacity (In other words, they didn’t ask them how does the OSS community processes this patch but asked them to process a patch themselves). The participants should have given consent to participate in the research and the risks of participating in it should have been disclosed, especially given the fact that missing a security bug and agreeing to merge it could be detrimental to someone’s reputation and future employability (that is, this would qualify for more than minimal risk for participants, requiring a full IRB review of the research design and process) with minimal benefits to them personally or to the organization as a whole (as it seems from the maintainers’ reaction to a new patch submission).

                                            One way to design this experiment ethically would have been to email the maintainers and invite them to participate in a “lab based” patch review process where the research team would present them with “good” and “bad” patches and ask them whether they would have accepted them or not. This is after they were informed about the study and exercised their right to informed consent. I really don’t see how emailing random stuff out and see how people interact with it (with their full name attached to it and in full view of their peers and employers) can qualify as research with less than minimal risks and that doesn’t involve human subjects.

                                            The other thing that rubs me the wrong way is that they sought (and supposedly received) retroactive IRB approval for this work. That wouldn’t fly with my IRB, as my IRB person would definitely rip me a new one for seeking retroactive IRB approval for work that is already done, data that was already collected, and a paper that is already written and submitted to a conference.

                                            1. 6

                                              You make excellent points.

                                              1. IRB review has to happen before the study is started. For NIH, the grant application has to have the IRB approval - even before a single experiment is even funded to be done, let alone actually done.
                                              2. I can see the value of doing a test “in the field” so as to get the natural state of the system. In a lab setting where the participants know they are being tested, various things will happen to skew results. The volunteer reviewers might be systematically different from the actual population of reviewers, the volunteers may be much more alert during the experiment and so on.

                                              The issue with this study is that there was no serious thought given to what are the ethical ramifications of this are.

                                              If the pen tested system has not asked to be pen tested then this is basically a criminal act. Otherwise all bank robbers could use the “I was just testing the security system” defense.

                                              1. 8

                                                The same requirement for prior IRB approval is necessary for NSF grants (which the authors seem to have received). By what they write in the paper and my interpretation of the circumstances, they self certified as conducting non-human research at time of submitting the grant and only asked their IRB for confirmation after they wrote the paper.

                                                Totally agree with the importance of “field experiment” work and that, sometimes, it is not possible to get prospective consent to participate in the research activities. However, the guidelines are clear on what activities fall within research activities that are exempt from prior consent. The only one that I think is applicable to this case is exception 3(ii):

                                                (ii) For the purpose of this provision, benign behavioral interventions are brief in duration, harmless, painless, not physically invasive, not likely to have a significant adverse lasting impact on the subjects, and the investigator has no reason to think the subjects will find the interventions offensive or embarrassing. Provided all such criteria are met, examples of such benign behavioral interventions would include having the subjects play an online game, having them solve puzzles under various noise conditions, or having them decide how to allocate a nominal amount of received cash between themselves and someone else.

                                                These usually cover “simple” psychology experiments involving mini games or economics games involving money.

                                                In the case of this kernel patching experiment, it is clear that this experiment doesn’t meet this requirement as participants have found this intervention offensive or embarrassing, to the point that they are banning the researchers’ institution from pushing patched to the kernel. Also, I am not sure if reviewing a patch is a “benign game” as this is the reviewers’ jobs, most likely. Plus, the patch review could have adverse lasting impact on the subject if they get asked to stop reviewing patches if they don’t catch the security risk (e.g., being deemed imcompetent).

                                                Moreover, there is this follow up stipulation:

                                                (iii) If the research involves deceiving the subjects regarding the nature or purposes of the research, this exemption is not applicable unless the subject authorizes the deception through a prospective agreement to participate in research in circumstances in which the subject is informed that he or she will be unaware of or misled regarding the nature or purposes of the research.

                                                As their patch submission process was deceptive in nature, as their outline in the paper, exemption 3(ii) cannot apply to this work unless they notify maintainers that they will be participating in a deceptive research study about kernel patching.

                                                That leaves the authors to either pursue full IRB review for their work (as a full IRB review can approve a deceptive research project if it deems it appropriate and the risk/benefit balance is in favor to the participants) or to self-certify as non-human subjects research and fix any problems later. They decided to go with the latter.

                                            2. 35

                                              We believe that an effective and immediate action would be to update the code of conduct of OSS, such as adding a term like “by submitting the patch, I agree to not intend to introduce bugs.”

                                              I copied this from that paper. This is not research, anyone who writes a sentence like this with a straight face is a complete moron and is just mocking about. I hope all of this will be reported to their university.

                                              1. 18

                                                It’s not human research because we don’t collect personal information

                                                I yelled bullshit so loud at this sentence that it woke up the neighbors’ dog.

                                                1. 2

                                                  Yeah, that came from the “clarifiactions” which is garbage top to bottom. They should have apologized, accepted the consequences and left it at that. Here’s another thing they came up with in that PDF:

                                                  Suggestions to improving the patching process In the paper, we provide our suggestions to improve the patching process.

                                                  • OSS projects would be suggested to update the code of conduct, something like “By submitting the patch, I agree to not intend to introduce bugs”

                                                  i.e. people should say they won’t do exactly what we did.

                                                  They acted in bad faith, skirted IRB through incompetence (let’s assume incompetence and not malice) and then act surprised.

                                                2. 14

                                                  Apparently they didn’t ask the IRB about the ethics of the research until the paper was already written: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf

                                                  Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. We apologize for the raised concerns. This is an important lesson we learned—Do not trust ourselves on determining human research; always refer to IRB whenever a study might be involving any human subjects in any form. We would like to thank the people who suggested us to talk to IRB after seeing the paper abstract.

                                                  1. 14

                                                    I don’t approve of researchers YOLOing IRB protocols, but I also want this research done. I’m sure many people here are cynical/realistic enough that the results of this study aren’t surprising. “Of course you can get malicious code in the kernel. What sweet summer child thought otherwise?” But the industry as a whole proceeds largely as if that’s not the case (or you could say that most actors have no ability to do anything about the problem). Heighten the contradictions!

                                                    There are some scary things in that thread. It sounds as if some of the malicious patches reached stable, which suggests that the author mostly failed by not being conservative enough in what they sent. Or for instance:

                                                    Right, my guess is that many maintainers failed in the trap when they saw respectful address @umn.edu together with commit message saying about “new static analyzer tool”.

                                                    1. 17

                                                      I agree, while this is totally unethical, it’s very important to know how good the review processes are. If one curious grad student at one university is trying it, you know every government intelligence department is trying it.

                                                      1. 8

                                                        I entirely agree that we need research on this topic. There’s better ways of doing it though. If there aren’t better ways of doing it, then it’s the researcher’s job to invent them.

                                                      2. 7

                                                        It sounds as if some of the malicious patches reached stable

                                                        Some patches from this University reached stable, but it’s not clear to me that those patches also introduced (intentional) vulnerabilities; the paper explicitly mentions the steps that they’re taking steps to ensure those patches don’t reach stable (I omitted that part, but it’s just before the part I cited)

                                                        All umn.edu are being reverted, but at this point it’s mostly a matter of “we don’t trust these patches and will need additional review” rather than “they introduced security vulnerabilities”. A number of patches already have replies from maintainers indicating they’re genuine and should not be reverted.

                                                        1. 5

                                                          Yes, whether actual security holes reached stable or not is not completely clear to me (or apparently to maintainers!). I got that impression from the thread, but it’s a little hard to say.

                                                          Since the supposed mechanism for keeping them from reaching stable is conscious effort on the part of the researchers to mitigate them, I think the point may still stand.

                                                          1. 1

                                                            It’s also hard to figure out what the case is since there is no clear answer what the commits where, and where they are.

                                                        2. 4

                                                          The Linux review process is so slow that it’s really common for downstream folks to grab under-review patches and run with them. It’s therefore incredibly irresponsible to put patches that you know introduce security vulnerabilities into this form. Saying ‘oh, well, we were going to tell people before they were deployed’ is not an excuse and I’d expect it to be a pretty clear-cut violation of the Computer Misuse Act here and equivalent local laws elsewhere. That’s ignoring the fact that they were running experiments on people without their consent.

                                                          I’m pretty appalled the Oakland accepted the paper for publication. I’ve seen paper rejected from there before because they didn’t have appropriate ethics review oversite.

                                                      1. 3

                                                        One simple (simplistic?) measurement of pure startup time for interpreters: https://gist.github.com//catwell/8189169.

                                                        1. 22

                                                          i think the most effective thing linux laptop producers could do to increase adoption is to improve the touchpad drivers and make it as nice an experience as Mac. As a non power user of the Mac specific software this was the biggest downside of other laptops I have tried.

                                                          1. 5

                                                            I second this, I use a Mac at work and an xps13 otherwise and this frustrates me so much.

                                                            The second point is wifi and Bluetooth stability that is completely unreliable compared to macOS…

                                                            1. -2

                                                              You use a touchpad when you’re at work?

                                                              1. 12

                                                                questions like this make me not want to participate in this website.

                                                                1. 3

                                                                  I use a Magic Trackpad on my stationary computer. It’s better for everything except precision pixel painting, especially for your body.

                                                                  1. 3

                                                                    Not the parent poster, but yes. I do a modest percentage of my work using a laptop as a laptop.

                                                                    In pre-pandemic times, I had frequent in person meetings. In pandemic times, I sometimes have to be where I can see/hear the kids.

                                                                    With a Mac, I just use the trackpad. It’s not something I’d do 8 hours a day, but I have zero problems using it when I’m mobile. On linux, I either schlep around a mouse, which is inconvenient, or use the trackpad, which is also frustrating.

                                                                    1. 2

                                                                      Many prefer to cyble between multiple mouse inputs to combat carpal tunnel and other strain related injuries. I much prefer a touch pad myself.

                                                                1. 23

                                                                  What good is a hardware with stellar performance and battery life if you are encouraged to get locked in a walled garden software ecosystem?

                                                                  Is it not worth making a tradeoff today and wait a year or two before non-Apple manufacturers started making ARM laptops with Linux support?

                                                                  1. 5

                                                                    Note that linux/windows laptops on ARM already exist, and are not competitive with Apple’s M1 macs in terms of performance. It’s certainly the case that Apple’s efficiency is some combination of their chip design and ARM. While I’m not super-well informed, I would say I think it’s Apple’s own work contributed more to the relative improvement over most laptops than ARM itself.

                                                                    1. 3

                                                                      What good is a hardware with stellar performance and battery life if you are encouraged to get locked in a walled garden software ecosystem?

                                                                      The software I use most often on the various laptops I have – both personal and employer-issued – is a list of web browsers which are cross-platform, (Chrome for work, Firefox for personal), chat apps which are cross-platform, developer tools that are either cross-platform (Emacs, Docker) or just platform-flavored versions of things that have equivalents elsewhere (terminal emulators, for example), the MS Office suite (not available on Linux), and then a mix of “whatever the platform’s utility app for this is” (mail client, music player, etc.).

                                                                      I’m not sure exactly how I am “locked into” anything by this. I also don’t have to go out of my way to try to avoid some sort of “walled garden”.

                                                                      And the other advantage, so often unspoken in these discussions, is that while with some companies I am merely a user of services/products, with Apple I generally am actually the customer, and the business consists of offering goods and services to me in exchange for money. Compared to business models where I am part of the product, with my usage subsidized by harvesting and selling great gobs of data about my habits, I literally could not care less that Apple runs an app store some people dislike.

                                                                      1. 1

                                                                        I have used all three major OSes for multiple years each, and easily find Linux to the most free of all (Windows comes second, thanks to WSL). I can easily run a server declaratively configured and have my development machine use the exact environment, for instance, which is what not being locked into a walled garden software ecosystem can naturally afford one.

                                                                        Outside of brand and marketing, Apple’s ecosystem holds very little appeal to me (and I used to champion it, whilst owning a Macbook Pro, iPhone, Apple Watch, AirPods and what not). As a hobbyist programmer, I found it beneficial in the long run to leave Apple’s garden.

                                                                        1. 4

                                                                          I used various flavors of Linux as my main desktop OS for quite a few years, starting around 2001. I don’t anymore.

                                                                          Partly this is because, at the end of the day/week, I no longer want to hack unless it’s on my personal projects. I especially don’t want to be doing it just to get my computer to work. But that was the Linux desktop treadmill: things are broken in different slightly-annoying ways, and there’s some asymptotic approach to non-brokenness, but it never gets there because there’s always a complete backwards-incompatible rewrite of the relevant software before it reaches actual polished usability.

                                                                          Meanwhile, the older I get the more I realize that the FSF/RMS threat model is not my threat model, and largely is not a relevant threat model anymore. The things I worry about and want to be protected from are things they don’t worry about and seem to think I deserve to have inflicted on me if ever I perform insufficient due diligence.

                                                                          Apple’s stuff works for me. Do they have an app store? Yes. Do I have to use it to get software onto my computer? No. Do the tradeoffs they’ve made strike me as reasonable in balancing the need to protect most non-technical people while making sure I can still get my work done? Yes.

                                                                          YMMV, but presenting this in the kind of absolute/objective terms you have does not work.

                                                                    1. 3

                                                                      I started working on advanced developer tools 9 years ago. Back when I started, “programming tools” meant file format viewers, editors, and maybe variants of grep.

                                                                      … and no, this is not taken out of context. This is the whole first paragraph.

                                                                      At least this post starts with wild hyperbole. Eclipse existed 9 years ago (as did a number of other tools), abd it goes a long way beyond this.

                                                                      1. 4

                                                                        … and no, this is not taken out of context. This is the whole first paragraph.

                                                                        Is Eclipse not an editor?

                                                                        Coverity also existed, but that’s not what people would think of when I said “dev tools.”

                                                                        1. 5

                                                                          I wouldn’t be as hard on you as the GP, but I feel like just dismissing Eclipse as an editor obscures more than it reveals. Aside from the whole IDE/editor terminology tarpit, Eclipse (and Intellij, which I’ve used more recently) include lots of tools for exploring, refactoring, visualizing and debugging code.

                                                                          I think all three tools you describe in your post sound like the kind of things you could conceivably consume as plugins in one of those IDEs (that could either be via some connection to an external tool, or by actually embedding it as plugin with no external dependencies).

                                                                          That quibble aside, I agree with your post. There’s a lot of interesting ideas coming out of academia that aren’t maintained, or even worse, never become usable. This may be a separate point, but even without an academic background, it’s pretty easy to find limitations of the existing tools as an industry programmer (e.g. IntelliJ structural search and replace is wonderful, but once you’re past what it can do, it gets painful quite quickly).

                                                                          1. 2

                                                                            Cool. I don’t think we have any disagreement, other than that you and GP think of a different cluster of tools when I gesture at “format viewers and editors.”

                                                                          2. 4

                                                                            It is/contains an editor, all IDEs do.

                                                                            But it is not just “file format viewers, editors, and maybe variants of grep”:

                                                                            There was intelligent refactoring, dependencies, debugging etc.

                                                                        1. 15

                                                                          I find it really odd that UUIDs didn’t come up here.

                                                                          They solve the same problems without the drawback that they can be accidentally used as temporary values when iterating or doing maths

                                                                          1. 17

                                                                            That’s a start, but using a common ID type besides integer or string only prevents some of the errors. You can still inadvertently pass a Person ID to an operation expecting a Product ID, unless they are distinct in your type system. I like to pair each table with a one-field struct type holding its ID value. Some languages make this zero cost at runtime for a nice compile time benefit.

                                                                            Types are very good when they give you guarantees about data, but even better when they give you guarantees about the problem domain.

                                                                            1. 14

                                                                              You can still inadvertently pass a Person ID to an operation expecting a Product ID, unless they are distinct in your type system

                                                                              you can, but if there actually is a collision, then you have much worse problems in your hands.

                                                                              1. 1

                                                                                Or a very (* huge number of times) unlikely, but not impossible, event happened once… either way, you’d have to be even more unlucky/have a bigger problem to collision within the same namespace.

                                                                                1. 1

                                                                                  It’s not either-or. You can make your types mutually exclusive at compile time and your values collision-resistant at runtime too.

                                                                                2. 3

                                                                                  That’s a good idea 💡! I’ve also seen the issue you mentioned solved by prepending the table name to the id “person:12345” or using opaque type aliases (if the language has a type system that supports this)

                                                                                  1. 5

                                                                                    Embedding the type along with the identifier is such a good idea that I’ve surprised myself with how much I used to scoff at this when looking at IDs on paper communications, e.g. ‘CST12345’ having a ‘CST’ prefix so that it’s unambiguously (for the organisation) a customer identifier rather than a widget or a data or … whatever.

                                                                                    Within a relational database, in X normal form tables, the ‘type’ of a row is effectively identified with the table name. With other forms of data storage, there are other approaches. With CSV, for example, column name serves this purpose. In a structured document, e.g. JSON, we might have:

                                                                                    "customer-id": 12345
                                                                                    

                                                                                    It’s still far too easy for confusion to happen, though.

                                                                                    In data storage, we might do this:

                                                                                    "customers": [
                                                                                        { "id": 42, name: "Alice", "ref": 99 },
                                                                                        { "id": 34, name: "Bob", "ref": 123 }
                                                                                    ]
                                                                                    

                                                                                    ref … to what? It’s something that has an ID, and that ID is stored as an integer, but:

                                                                                    1. We don’t know what it is, so it’s easy to make a wrong assumption.
                                                                                    2. It’s an integer, so if we get it wrong, we might not even notice until catastrophe occurs.

                                                                                    In code, too, we can do (pseudocode here):

                                                                                    function update_customer(id, ref, from_date):
                                                                                        save(id = id, address_ref = ref, from_date = from_date)
                                                                                    

                                                                                    Again here we’ve made an assumption, and if new_ref wasn’t supposed to be an address_ref, we have a catastrophe. Instead of updating when the customer started the job with ref ‘ref’, we’ve set when they moved into the address given by ‘ref’, which was not the intention.

                                                                                    Does this mean that we should lean on the type system? In some languages, this works really well. Apologies because I haven’t used C++ for a long time, but I seem to remember you can do this:

                                                                                    void update_customer(Customer customer, Job job, Date from_date) {
                                                                                        save_job_details(customer.id, job.id, from_date);
                                                                                    }
                                                                                    

                                                                                    Here we’ve fixed the issue of understanding what we’ve been passed, but there’s still a boundary here between the code and the underlying data, where we can easily get things confused, because the data is stored without strict types - customer ID is stored as a number, job ID is stored as a number - even date is (let’s assume here) stored as a number - and they could be switched around without the code or the data storage caring.

                                                                                    We could have a storage system that enforced the types in the same way as the type system we’ve built in the code, but I don’t think I’ve ever used a system like this. Once I’m working with data storage or transmission, I’m mapping to/from a structure where the highest level enforced types are only as sophisticated as integer vs bigger integer vs date vs text, for example.

                                                                                    We’ve gone some way to helping ourselves avoid a particular kind of catastrophe by using GUID/UUIDs as unique identifiers, so even if we get confused, at least the chance of mistaken identity is effectively zero.

                                                                                    Why doesn’t an XML/JSON schema fix the problem? Because it only works as well as the mapping code. I’ve noticed there’s been a trend away from schemas as JSON’s become the lingua franca. I used to really like them in the XML world because they provided a little safety ‘for free’ and then building some checks on top was much less work.

                                                                                    Why don’t ORMs solve the problem? They do help, but I think we’ve all seen plenty of positive and negative aspects of using ORMs, so they aren’t ubiquitous, especially in the case where they are hooked into the types through application code, and of course the world isn’t using relational databases for everything any more.

                                                                                    Back to labelling identifiers with their types: It’s a safety belt, it adds boiler plate, and we often don’t like ‘unnecessary’ fluff, but with a little tooling support, perhaps it’s one of the better options for avoiding type confusion that we have when it comes to data leaving the confines of code using high level types for safety.

                                                                                3. 13

                                                                                  They don’t only solve solve that problem. They solve the problem of generating random numbers on a distributed manner without collisions. And because of this they are a rather huge number, which makes them a very bad choice for database index.

                                                                                  I don’t quite agree with this article, in the sense that the examples provided by the author are pretty well known anti patterns. The reason why you probably shouldn’t do it is that you shouldn’t do it at all. An incremental integer primary key is not meant to be used as a public reference for external systems. If you need another unique identifier, add one as needed. You can generate a large random integer for example. I don’t think anyone would write a for loop and use a loop counter for database lookup. The point of having a database is to NOT do such things.

                                                                                  1. 7

                                                                                    I’ve mapped “UUIDs are a bad idea for database keys” to a “needs more research” thunk in my head. I have no idea if it’s true or not. One place it was discussed: https://news.ycombinator.com/item?id=14523523.

                                                                                    1. 4

                                                                                      They don’t only solve solve that problem. They solve the problem of generating random numbers on a distributed manner without collisions. And because of this they are a rather huge number, which makes them a very bad choice for database index.

                                                                                      Why? Indexes aren’t using the integers directly. Binary search – and b-trees – don’t care how big a value is. Neither do hash indexes. UUIDs aren’t ordered, but I doubt that user ids would have a huge locality effect – or that you’d have enough users that the index pages would fall out of cache.

                                                                                      What’s the concern? Storage size?

                                                                                      1. 2

                                                                                        So from personal experience I worked at a place using MSSQL and they had GUIDs as the Clustered Index (CI). Now aside from the fact that these were stored as the wrong type (MSSQL has a UUID type which should have been able to handle them a bit more space efficiently) the other problem was table bloating.

                                                                                        See, when you have an ordered index and you insert a user into the table it just gets appended to the end of the table in the last page. Then the indexes get updated and those might end up having to be inserted in the middle of a page somewhere in the middle of the index but because indexes are usually smaller than table records this isn’t a massive deal.

                                                                                        Now if your clustered index isn’t ordered you end eventually with an incredibly sparse table, because if you consider a completely defragmented and compacted table, if you need to perform an insert this insert is going to be guaranteed to be in the middle of the table somewhere in the middle of a page. This causes a page split (which means you’ve now got two pages). This keeps happening until your tables all take up twice as much space.

                                                                                        And when your key type is a 36 byte fixed string and everything refers to everything and your keys end up included as index elements for various reasons this means that your database is now growing at an enormous rate all the time.

                                                                                        It wasn’t a pretty picture especially since we were trying to use the free tier of MSSQL which has a 10GiB per/db limit.

                                                                                        When the solution of splitting the database up into multiple databases (which brought in a whole slew of problems) was eventually implemented after many of my protests we tried to change out all the CIs in the database. GUIDs would still be used for references but an integer RowID was being used as the CI.

                                                                                        Databases grew slower and things actually performed better.

                                                                                        Although the above is an anecdote.

                                                                                    2. 8

                                                                                      When running my code in ‘test mode’, I adjusted the autoincrement sequences in the schema so that none of them generate overlapping values (so that if I use the wrong table ID I can’t accidentally find a different record). Not a perfect system, but a very cheap one to implement that catches 90% of my mistakes.

                                                                                      1. 1

                                                                                        Indeed, this is a great idea to catch more bugs in an existing system that is using integer IDs!

                                                                                      2. 4

                                                                                        Or you simply use a UserId for the the user id and be done, because it completely solves the issue described in the article (hint: it’s not a collision/uniqueness problem).

                                                                                        1. 1

                                                                                          hint: it’s not a collision/uniqueness problem

                                                                                          Exactly. UUIDs aren’t a bad solution, but they aren’t targeting the actual problem being discussed, even if they happen to work well for it.

                                                                                      1. 3

                                                                                        My perspective is that the less likely it is that a language is any given user’s primary tool, or at least essential for their projects, the more important it is that the libraries and tooling not get in their way. Haskell probably suffers less than some languages, because its reputation warns people that the language itself requires a lot of work. Still, I think reducing friction is more important than almost anyone realizes. As such, I think changing base is by far the best solution, unless it’s truly impossible.

                                                                                        I’m familiar with alternate preludes, but I never used one because I didn’t want to have to make a choice. Selecting an alternate prelude feels like a bet on the library being maintained and widely used. If there’s a blessed choice that the community rallies around, it still becomes one more hurdle to getting started. If there isn’t, then that’s even worse.

                                                                                        Background: I have a few thousand lines of Haskell code written a few years ago that I still use on a regular basis, but I go months without touching Haskell. I consider myself an advanced beginner.

                                                                                          1. 4

                                                                                            https://twitter.com/FredericJacobs/status/1367115794363088897

                                                                                            I think it’s safe to say we’re reaching the middle of the confusion.

                                                                                          1. 10

                                                                                            I use Microsoft Sculpt Ergonomic keyboard.

                                                                                            I was using the Mac keyboard beforehand, but my left wrist + ligaments on top of the hand started hurting after prolonged usage, so I knew I had to switch to an ergonomic keyboard. One of the best purchases of my life. Tbh, I rly dislike the fact that the keyboard is connected thru USB instead of bluetooth, but I assume that’s some business decision, in order to force pairing the keyboard with the mouse (cuz you can’t buy the keyboard standalone… it comes with the mouse).

                                                                                            1. 2

                                                                                              I also use this keyboard on a Mac and I absolutely love it.

                                                                                              1. 2

                                                                                                I use the Microsoft Sculpt Ergonomic keyboard too. I love it. It feels great to type on even after long periods. It’s not huge, it’s light and easy to move around my desk. It’s not mechanical unfortunately. I also wish it was Bluetooth. But honestly, the worst part for me is whatever material they used for the palm rest wears away relatively quickly. I’m already on my 2nd keyboard and thinking about buying another one. I might buy a bunch because Microsoft will stop making them at some point and I don’t want to pay a fortune in the post-production market.

                                                                                                1. 4

                                                                                                  But honestly, the worst part for me is whatever material they used for the palm rest wears away relatively quickly

                                                                                                  I’ve had to take apart my 1st to try and repair it, but failed. I noticed though while taking it apart, it is VERY easy to replace this wrist rest material with something of your own. It would be a easy modification to use another material.

                                                                                                  I think a big reason behind the material they chose is it’s easy to clean while still being soft.

                                                                                                  1. 2

                                                                                                    Can you explain? I didn’t think the wrist rest would be that easy to replace, mostly because of the shape and how it’s attached to the rest of the plastic. What material do you recommend and how would you get everything fit in nicely?

                                                                                                    1. 3

                                                                                                      You wouldn’t replace the rest but the material. The material just slides right off :)

                                                                                                      1. 2

                                                                                                        I get it now. Very interesting. I don’t think it will be easy but peeling off just the stop surface is more promising than that what I was originally picturing. Thanks.

                                                                                                2. 2

                                                                                                  You used to be able to buy one without a mouse, and it still used the dongle.

                                                                                                  It’s a mixed bag: I lost the dongle to one keyboard, which sucked. On the other hand, my experience (which I’ve heard echoed from others) is that wireless dongles connect much more reliably than Bluetooth.

                                                                                                  1. 2

                                                                                                    I use the old-style Microsoft Ergonomic Pro keyboards - produced when they were switching from PS/2 connectors to USB, so they come with a split connector with both. Right now I troll eBay for ones that look like they’re in good shape (haven’t found new in box for sale) because there hasn’t been a good replacement made since the early 2000s. The Logitech ergonomic keyboards from early 2000s were good, but it seems Logitech got out of producing real ergo keyboards.

                                                                                                    1. 2

                                                                                                      It’s a shame they don’t make a slightly better quality one, as the ergonomics is just perfect. Plus, it doesn’t differ so much from a regular QWERTY keyboard.

                                                                                                      However, some keys get stuck, latency is a bit too high for my taste and the rubber palm rest gets damaged too quickly.

                                                                                                      1. 2

                                                                                                        Thanks for the recommendation! Btw, seems like you can indeed buy a keyboard alone: https://www.amazon.de/Ergonomische-Tastatur-Microsoft-Sculpt-QWERTZ-Layout/dp/B00EY8PXZG/

                                                                                                      1. 3

                                                                                                        I think abstraction actually means like six different things and a lot of discussions about abstraction are because people are conflating some of these things. Kinda like “naming things”.

                                                                                                        EDIT: literally the exact thing this article is about, I didn’t read it well enough

                                                                                                        1. 3

                                                                                                          I’m not remotely surprised you have identified more buckets than I have. Can I ask what they are?

                                                                                                          1. 2

                                                                                                            I haven’t actually identified any buckets! It’s just the way the discussion “tastes” to me, you know? Like reading arguments about it fall into that kind of template where it seems everybody is thinking about something different when they use abstraction. I’ll think on this more and get back to you with some possible differences.

                                                                                                            EDIT: oh my god, this is basically the exact same point you made in the article. I really shouldn’t be trying to write hot takes while sleep deprived. Sorry.

                                                                                                        1. 74

                                                                                                          First, their argument for Rust (and against C) because of memory safety implies that they have not done due diligence in finding and fixing such bugs. […] And with my bc, I did my due diligence with memory safety. I fuzzed my bc and eliminated all of the bugs.

                                                                                                          This seems like such a short-sighted, limited view. Software is not bug-free. And ruling out a class of bugs by choice of technology as a measure of improving overall robustness won’t fix everything, but at the very least it’s a trade-off that deserves more thorough analysis than this empty dismissal.

                                                                                                          1. 21

                                                                                                            I think he is probably right (after all, he wrote it) when he says rewriting his bc in Rust would make it more buggy. I disagree this is an empty dismissal, since it is backed by his personal experience.

                                                                                                            For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.

                                                                                                            1. 15

                                                                                                              I think there’s a big difference between programs and libraries with stable requirements and those that evolve here. The bc utility is basically doing the same thing that it did 20 years ago. It has a spec defined by POSIX and a few extensions. There is little need to modify it other than to fix bugs. It occasionally gets new features, but they’re small incremental changes.

                                                                                                              Any decision to rewrite a project is a trade off between the benefits from fixing the accumulated technical debt and the cost of doing and validating the rewrite. For something stable with little need of future changes, that trade is easy to see: the cost of the rewrite is high, the benefit is low. In terms of rewriting in a memory-safe language, there’s an additional trade between the cost of a memory safety vulnerability and the cost of the rewrite. The cost of Heartbleed in OpenSSL was phenomenal, significantly higher than the cost of rewriting the crypto library. In the cast of bc, the cost of a memory safety bug is pretty negligible.

                                                                                                              Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs. The author talks about fuzzing, address sanitiser, and so on. These are great tools. They’re also completely unnecessary in a memory-safe language because they try to find classes of bugs that you cannot introduce in the first place in a memory-safe language (and they do so probabilistically, never guaranteeing that they’ve found them all).

                                                                                                              If you’re starting a new project, then you need a really good reason to start it in C and pay the cost of all of that fuzzing.

                                                                                                              1. 17

                                                                                                                Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs.

                                                                                                                There’s an implied assumption here that if a language is memory safe, those memory safe bugs will simply go away. In my experience, that is not quite true. Sometimes those memory safety bugs will turn into logic bugs.

                                                                                                                Not to pick on Rust here, but in Rust it is very common to put values into an array and use array indices instead of pointers when you have some kind of self-referential data structure that’s impossible to express otherwise using rust’s move semantics. If you simply do such a naive transformation of your C algorithm, your code will be memory safe, but all your bugs, use after free, etc, will still be there. You just lifted them to logical bugs.

                                                                                                                Rust has no good abstractions to deal with this problem, there are some attempts but they all have various practical problems.

                                                                                                                Other languages like ATS and F* have abstractions to help with this problem directly, as well as other problems of logical soundness.

                                                                                                                1. 13

                                                                                                                  Right - but in lifting these from memory bugs to logic bugs, you get a runtime panic/abort instead of a jump to a (likely-attacker-controllable) address. That’s a very different kind of impact!

                                                                                                                  1. 9

                                                                                                                    You don’t get a panic if you access the “wrong” array index. The index is still a valid index for the array. Its meaning (allocated slot, free slot, etc), is lost to the type system, though in a more advanced language it need not be. This later leads to data corruption, etc, just like in C.

                                                                                                                    1. 7

                                                                                                                      It leads to a much safer variant of data corruption though. Instead of corrupting arbitrary memory in c or c++ (like a function pointer, vtable, or return address), you are only corrupting a single variable’s value in allocated and valid and aligned memory (like a single int).

                                                                                                                      You would get a panic in rust for every memory corruption bug that could cause arbitrary code execution, which is what matters.

                                                                                                                      1. 1

                                                                                                                        This later leads to data corruption, etc, just like in C.

                                                                                                                        Can you expand on this? I had expected the behaviour in Rust to be significantly safer than C here. In C, the data corruption caused by use-after free often allows an attacker to execute arbitrary code.

                                                                                                                        I totally see your point about logical corruption (including things like exposing critical secrets), but I don’t follow that all the way to “just like C”. How would an array index error be exploited in Rust to execute arbitrary code?

                                                                                                                        1. 11

                                                                                                                          I have once written a bytecode interpreter in C++, for a garbage collected scripting language. I implemented my own two-space garbage collector. For performance reasons, I didn’t use malloc() directly, but instead allocated a big enough byte array to host all my things. If I overflew that array, Valgrind could see it. But if I messed up it’s internal structure, no dice. That heap of mine was full of indices and sizes, and I made many mistakes that caused them to be corrupted, or somehow not quite right. And I had no way to tell.

                                                                                                                          I solved this by writing my own custom heap analyser, that examined the byte array and tell me what’s in there. If I see all my “allocated” objects in order, all was well. Often, I would see something was amiss, and I could go and fix the bug. Had I written it in Rust instead, I would have had to write the exact same custom heap analyser. Because Rust wouldn’t have prevented me from putting the wrong values inside my array. It’s perfectly “safe” after all, to write gibberish in that array as long as I don’t overflow it.

                                                                                                                          Now could this particular bug lead to arbitrary code execution? Well, not quite. It would generate wrong results, but it would only execute what my C++/Rust program would normally execute. In this case however, I was implementing a freaking scripting language. The code an attacker could execute wasn’t quite arbitrary, but it came pretty damn close.

                                                                                                                          1. 6

                                                                                                                            The effects of data corruption depend on what the code does with the data. This often means arbitrary code execution, but not always. It’s not a property of C, it’s a property of the code. This doesn’t change when you change the implementation language.

                                                                                                                            Fundamentally there is no semantic difference between a pointer in a C heap and an array index into a Rust array. In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code. It’s what the code does with the data that leads to exploitation (or not).

                                                                                                                            Of course Rust has many additional safety advantages compared to C, buffer overflows don’t smash the stack, etc, and using references in Rust if you can is safe. And when using references, there’s a great deal of correlation between Rust’s notion of memory safety and true logic safety. This is good! But many people don’t realise that this safety is predicated on the lack of aliasing. The borrow checker is only a mechanism to enforce this invariant, it’s not an operative abstraction. It’s the lack of aliasing that gets you the safety, not the borrow checker itself. When you give up aliasing, you lose a lot of what Rust can do for you. Virtually everybody understands that if you introduce unsafe pointers, they give up safety, but less people seem to understand that introducing aliasing via otherwise safe mechanism has the same effect. Of course, the program continues to be memory safe in Rust terms, but you lose the strong correlation between memory safety and logic safety that you used to have.

                                                                                                                            Not that there’s anything wrong with this, mind you, it’s just something people need to be aware of, just as they are already aware of the tradeoffs that they make when using unsafe. It does make a projection for the number of bugs that Rust can prevent in practice more difficult, though.

                                                                                                                            1. 6

                                                                                                                              I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”. Even a rust alias logic bug dies not open up this kind of arbitrary code execution exposure because you can’t alias an int with a function pointer or a vtable or a return address on the stack, like you can in c or c++. You can only alias an int with an int in safe rust, which is an order of magnitude safer and really does eliminate an entire class of vulnerabilities.

                                                                                                                              1. 6

                                                                                                                                I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”.

                                                                                                                                In the security research world, we usually treat control of the program counter (the aptly named rip on x86-64) as “arbitrary code execution.” You can do a surprising of programming using code that’s already in a process without sending any byte code of your own with return-oriented programming.

                                                                                                                                1. 3

                                                                                                                                  But does Rust let you do that here? What does a snippet of Rust code look like that allows attacker-controlled indexing into an array escalate to controlling the program counter?

                                                                                                                                  1. 2

                                                                                                                                    Surely you agree that “variables changing underfoot” implies “programs flow becomes different from what I expect”. That’s why we use variables, to hold the Turing machine state which influences the next state. A logical use after free means “variables changing underfoot”. You don’t expect a free array slot’s value (perhaps now reallocated) to change based on some remote code, but it does.

                                                                                                                                    1. 3

                                                                                                                                      Right, but “program flow becomes different from what I expect, but it still must flow only to instruction sequences that the original program encoded” is much much safer than “program flow can be pointed at arbitrary memory, which might not even contain instructions, or might contain user-supplied data”.

                                                                                                                                      1. 2

                                                                                                                                        With ROP, the program flow only goes through “instruction sequences that the original program encoded”, and yet ROP is pretty much fatal.

                                                                                                                                        1. 7

                                                                                                                                          ROP is not possible when you index an array wrong in rust, what is your point?

                                                                                                                                          1. 6

                                                                                                                                            And you can’t do rop in safe rust.

                                                                                                                                            1. 2

                                                                                                                                              Maybe not directly within the native code of the program itself, but I think (at least part of) 4ad’s point is that that’s not the only level of abstraction that matters (the memory bug vs. logic bug distinction).

                                                                                                                                              As an example, consider a CPU emulator written entirely in safe Rust that indexes into a u8 array to perform its emulated memory accesses. If you compile an unsafe program to whatever ISA you’re emulating and execute it on your emulator, a bad input could still lead to arbitrary code execution – it’s at the next semantic level up and not at the level of your program itself, but how much does that ultimately matter? (It’s not really terribly different than ROP – attacker-controlled inputs determining what parts of your program get executed.)

                                                                                                                                              That’s admittedly a somewhat “extreme” case, but I don’t think the distinction between programs that do fall into that category and those that don’t is terribly clear. Nearly any program can, if you squint a bit, be viewed essentially as a specialized interpreter for the language of its config file (or command-line flags or whatever else).

                                                                                                                                              1. 2

                                                                                                                                                There’s no distinction here. If your program implements a cpu emulator then your program can execute with no arbitrary code execution at all and still emulate arbitrary code execution on the virtual cpu. If you want the virtual program executing to not possibly execute arbitrary virtual instructions, you need to generate the virtual program’s instructions using a safe language too.

                                                                                                                                                In most cases, though, arbitrary virtual code execution is less dangerous than arbitrary native code execution, though that’s beside the point.

                                                                                                                                                1. 2

                                                                                                                                                  So…we agree? My point was basically that attacker-controlled arbitrary code execution can happen at multiple semantic levels – in the emulator or in the emulated program (in my example), and writing the emulator in a safe language only protects against the former, while the latter can really be just as bad.

                                                                                                                                                  Though I realize now my example was poorly chosen, so a hopefully better one: even if both the emulator and the emulated program are written in memory-safe languages, if the emulator has a bug due to an array-index use-after-free that causes it to misbehave and incorrectly change the value of some byte of emulated memory, that destroys the safety guarantees of the emulated program and we’re back in arbitrary-badness-land.

                                                                                                                                                  1. 1

                                                                                                                                                    Sure but this is just as meaningful as talking about a cpu hardware bug that might cause a native safe program to run amok. Technically true but not very useful when evaluating the safe programming language

                                                                                                                                    2. 3

                                                                                                                                      Right, I agree, and safe rust aliasing that the GP described is not possible to control the program counter arbitrarily.

                                                                                                                                    3. 4

                                                                                                                                      Yeah exactly, this is the part I thought @4ad was arguing was possible. Eg. in C, use-after-free often allows me to make the program start interpreting attacker-provided data as machine code. I thought this is what 4ad was saying was also possible in Rust, but I don’t think that’s what they are claiming now.

                                                                                                                                      To me, that’s a big difference. Restricting the possible actions of a program to only those APIs and activities the original code includes, vs C where any machine code can be injected in this same scenario, is a major reduction in attack surface, to me.

                                                                                                                                      1. 4

                                                                                                                                        One thing to note is that code is data and data is code, in a true, hard-mathematical sense.

                                                                                                                                        The set of

                                                                                                                                        the possible actions of a program to only those APIs and activities the original code includes,

                                                                                                                                        and

                                                                                                                                        C where any machine code can be injected in this same scenario

                                                                                                                                        is exactly the same (unbounded!). Of course it is much easier in practice to effect desired behavior when you can inject shell code into programs, but that’s hardly required. You don’t need to inject code with ROP either (of course ROP itself is not possible in Rust because of other mitigations, this is just an example).

                                                                                                                                        Please note that in no way I am suggesting that Rust is doing anything bad here. Rust is raising the bar, which is great. I want the bar raised even higher, and we know for a fact that this is possible today both in theory and practice. Until we raise the bar, I want people to understand why we need to raise the bar.

                                                                                                                                        At the end of a day you either are type safe or you aren’t. Of course the specifics of what happens when you aren’t type safe depend on the language!

                                                                                                                                        PS: arrays can contain other things than integers, e.g. they can contain function pointers. Of course you can’t confuse an int with a function pointer, but using the wrong function pointer is pretty catastrophic.

                                                                                                                                        1. 3

                                                                                                                                          is exactly the same (unbounded!).

                                                                                                                                          I guess this is what I don’t understand, sorry for being dense. Can you show a concrete code example?

                                                                                                                                          In my mind I see a program like this:

                                                                                                                                          
                                                                                                                                          enum Action {
                                                                                                                                              GENERATE_USER_WEEKLY_REPORT,
                                                                                                                                              GENERATE_USER_DAILY_REPORT,
                                                                                                                                              LAUNCH_NUCLEAR_MISSILES
                                                                                                                                          }
                                                                                                                                          
                                                                                                                                          impl Action {
                                                                                                                                            pub fn run(&self) {
                                                                                                                                              ...
                                                                                                                                            }
                                                                                                                                          }
                                                                                                                                          
                                                                                                                                          // Remember to remove the nuclear missile action before calling!
                                                                                                                                          fn exploitable( my_actions:  &Vec<Box<Action>>, user_controlled: usize ) {
                                                                                                                                            my_actions[user_controlled].run();
                                                                                                                                          }
                                                                                                                                          
                                                                                                                                          

                                                                                                                                          In my mind, there are two differences between this code in Rust and similar code in C:

                                                                                                                                          1. This only allows the user to launch nuclear missiles; it does not allow them to, say, write to the harddrive or make network calls (unless one of the actions contained code that did that ofc); in C, I’d likely be able to make something like this call any system function I wanted to, whether machine code to do that was present in the original binary or not.

                                                                                                                                          2. In Rust, this doesn’t allow arbitrary control flow, I can’t make this jump to any function in the binary, I can only trick it into running the wrong Action; in C, I can call run on any arbitrary object anywhere in the heap.

                                                                                                                                          ie. in C, this would let me execute anything in the binary, while in Rust it still has to abide by the control flow of the original program, I thought was the case, anyway.

                                                                                                                                          I think you’re saying this is wrong, can you explain how/why and maybe show a code example if you can spare the time?

                                                                                                                                          1. 4

                                                                                                                                            This is correct and 4ad is mistaken. I’m not sure why 4ad believes the two are equivalent; they aren’t.

                                                                                                                                          2. 3

                                                                                                                                            “is exactly the same”

                                                                                                                                            It simply isn’t, and I’m not sure why you think it is.

                                                                                                                                      2. 1

                                                                                                                                        In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code.

                                                                                                                                        Do you have any links on this that you recommend?

                                                                                                                              2. 2

                                                                                                                                Good analysis. You didn’t use the words, but this is a great description of the distinction between stocks and flows: https://en.wikipedia.org/wiki/Stock_and_flow. I wish more people talking about software paid attention to it.

                                                                                                                                1. 2

                                                                                                                                  Author here.

                                                                                                                                  I would also argue that crypto should not change often, like bc. You might add ciphers, or deprecate old ones, but once a cipher is written and tested, there should be very little need for it to change. In my opinion.

                                                                                                                                2. 8

                                                                                                                                  For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy.

                                                                                                                                  Have they actually rewrote anything? Or have they instead selected a different crypto library they trust better than the previous one? On the one hand, Rust has no advantage over C in this particular context. On the other hand, they may have other reasons to trust the Rust library better than the C one. Maybe it’s better tested, or more widely used, or audited by more reputable companies.

                                                                                                                                  If I take your word for it however, I have to disagree. Rewriting a cryptographic library in Rust is more likely to introduce new bugs, than it is to fix bugs that haven’t already been found and fixed in the C code. I do think however that the risk is slim, if they take care to also port the entire test suite as well.

                                                                                                                                  1. 7

                                                                                                                                    In the Cryptography case isn’t the Rust addition some ASN.1 parsing code? This is cryptography adjacent but very much not the kind of different that your point about cryptography code makes. Parsing code unless it is very trivial and maybe not even then tends to be some of the more dangerous code you can write. In this particular case Rust is likely a large improvement in both ergonomics for the parsing as well as safety.

                                                                                                                                    1. 1

                                                                                                                                      You’ve got a point. I can weaken it somewhat, but not entirely eliminate it.

                                                                                                                                      I don’t consider ASN.1 “modern”. It’s over complicated for no good reason. Certificates can be much, much simpler than that: at each level, you have a public key, ID & expiration date, a certificate of the CA, and a signature from the CA. Just put them all in binary blobs, and the only thing left to parse are the ID & expiration date, which can be left to the application. And if the ID is an URL, and the expiration date is an 64-bit int representing seconds from epoch, there won’t be much parsing to do… Simply put, parsing certificate can be “very trivial”.

                                                                                                                                      Another angle is that if you need ASN.1 certificates, then you are almost certainly using TLS, so you’re probably in a context where you can afford the reduced portability of a safer language. Do use the safer language in this case.

                                                                                                                                      Yet another angle is that in practice, we can separate the parsing code from the rest of the cryptographic library. In my opinion, parsing of certificate formats do not belong to a low-level cryptographic library. In general, I believe the whole thing should be organised in tiers:

                                                                                                                                      • At the lowest level, you have the implementation of the cryptographic primitives.
                                                                                                                                      • Just above that, you have constructions: authenticated encryption, authenticated key exchange, PAKE…
                                                                                                                                      • Higher up still, you have file format, network packet formats, and certificates. They can (and should) still be trivial enough that even C can be trusted with them. They can still be implemented with zero dependencies, so C’s portability can still be a win. Though at that level, you probably have an idea of the target platforms, making portability less of a problem.
                                                                                                                                      • Higher up still is interfacing with the actual system: getting random numbers, talking to the file system, actually sending & receiving network packets… At that level, you definitely know which set of platforms you are targetting, and memory management & concurrency start becoming real issues. At that point you should seriously consider switching to a non-C, safer language.
                                                                                                                                      • At the highest level (the application), you should have switched away from C in almost all cases.
                                                                                                                                  2. 2

                                                                                                                                    For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.

                                                                                                                                    This is a fair point. When it comes down to it, whether I am right or wrong about it will only be seen in the consequences of the decision that they made.

                                                                                                                                  3. 14

                                                                                                                                    Here’s the more thorough analysis you’re asking for: this is cryptographic code we’re talking about. Many assumptions that would be reasonable for application code simply does not apply here:

                                                                                                                                    • Cryptographic code is pathologically straight-line, with very few branches.
                                                                                                                                    • Cryptographic code has pathologically simple allocation patterns. It often avoids heap allocation altogether.
                                                                                                                                    • Cryptographic code is pathogenically easy to test, because it is generally constant time: we can test all code paths by covering all possible input & output lengths. If it passes the sanitizers & valgrind under those conditions, it is almost certainly correct (with very few exceptions).

                                                                                                                                    I wrote a crypto library, and the worst bug it ever had wasn’t caused by C, but by a logic error that would have happened even in Haskell. What little undefined behaviour it did have didn’t have any visible effect on the generated code.

                                                                                                                                    Assuming you have a proper test suite (that tests all input & output lengths), and run that test suite with sanitisers & Valgrind, the kind of bug Rust fixes won’t occur in your cryptographic C code to begin with. There is therefore no practical advantage, in this particular case to using Rust over C. Especially when the target language is Python: you have to write bindings anyway, so you can’t really take advantage of Rust’s better APIs.

                                                                                                                                    1. 2

                                                                                                                                      These bugs still occur in critical software frequently. It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion), and the safer language guarantees more than your suggestions would. And there’s also no risk of someone forgetting to run those things.

                                                                                                                                      1. 6

                                                                                                                                        These bugs still occur in critical software frequently.

                                                                                                                                        Yes they do. I was specifically talking about one particular kind of critical software: cryptographic code. It’s a very narrow niche.

                                                                                                                                        It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion)

                                                                                                                                        In my 4 years of first hand experience writing cryptographic code, it’s really not. Rust needs the same test suite as C does, and turning on the sanitizers (or Valgrind) on this test suite is a command line away. The real advantage of Rust lies in its safer API (where you can give bounded buffers instead of raw pointers). Also, the rest of the application will almost certainly be much safer if it’s written in Rust instead of C.

                                                                                                                                        And there’s also no risk of someone forgetting to run those things.

                                                                                                                                        Someone who might forget those things has no business writing cryptographic code at all yet, be it in C or in Rust. (Note: when I started out, I had no business writing cryptographic code either. It took over 6 months of people findings bugs and me learning to write a better test suite before I could reasonably say my code was “production worthy”.)

                                                                                                                                        1. 6

                                                                                                                                          Rusts advantage goes much further than at the api boundary, but again the discussion should be around how to get safer languages more widely used (ergonomics, platform support) and not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”. To me it is like saying “super careful contractors with perfect tools don’t need safety gear”, except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?

                                                                                                                                          1. 4

                                                                                                                                            Rusts advantage goes much further than at the api boundary

                                                                                                                                            Yes it does. In almost all domains. I’m talking about modern cryptographic code.

                                                                                                                                            again the discussion should be around how to get safer languages more widely used (ergonomics, platform support)

                                                                                                                                            Write a spec. A formal one if possible. Then implement that spec for more platforms. Convincing projects to Rewrite It In Rust may work as a way to coerce people into supporting more platforms, but it also antagonises users who just get non-working software; such a strategy may not be optimal.

                                                                                                                                            not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”.

                                                                                                                                            You’re not hearing me. I’m not talking in general, I’m talking about the specific case of cryptographic code (I know, I’m repeating myself.)

                                                                                                                                            • In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.
                                                                                                                                            • In this specific case, Rust is not safer.
                                                                                                                                            • In this specific case, you need that perfect test suite. In either language.
                                                                                                                                            • In this specific case, you can write that perfect test suite. In either language.

                                                                                                                                            except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?

                                                                                                                                            I really don’t. I root out potential mistakes by expanding my test suite as soon as I learn about a new class of bugs. And as it happens, I am painfully aware of the mistakes I made. One of them was even a critical vulnerability. And you know what? Rust wouldn’t have saved me.

                                                                                                                                            Here are the bugs that Rust would have prevented:

                                                                                                                                            • An integer overflow that makes elliptic curves unusable on 16-bit platforms. Inconvenient, but (i) it’s not a vulnerability, and (ii) Monocypher’s elliptic curve code is poorly suited to 16-bit platforms (where I recommend C25519 instead).
                                                                                                                                            • An instance of undefined behaviour the sanitizers didn’t catch, that generated correct code on the compilers I could test. (Note that TweetNaCl itself also have a couple instances of undefined behaviour, which to my knowledge never caused anyone any problem so far. Undefined behaviour is unclean, but it’s not always a death sentence.)
                                                                                                                                            • A failure to compile code that relied on conditional compilation. I expect Rust has better ways than #ifdef, though I don’t actually know.

                                                                                                                                            Here are the bugs that Rust would not have prevented:

                                                                                                                                            • Failure to wipe internal buffers (a “best effort” attempt to erase secrets from the computer’s RAM).
                                                                                                                                            • A critical vulnerability where fake signatures are accepted as if they were genuine.

                                                                                                                                            Lesson learned: in this specific case, Rust would have prevented the unimportant bugs, and would have let the important ones slip through the cracks.

                                                                                                                                            1. 8

                                                                                                                                              I’m talking about modern cryptographic code.

                                                                                                                                              In this discussion, I think it is important to remind that cryptography developers are explicitly and intentionally not writing modern cryptographic code. One thing they want to use Rust on is ASN.1 parsing. Modern cryptographic practice is that you shouldn’t use ASN.1 and they are right. Implementing ASN.1 in Rust is also right.

                                                                                                                                              1. 4

                                                                                                                                                I’m talking about modern cryptographic code.

                                                                                                                                                So am I.

                                                                                                                                                In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.

                                                                                                                                                I disagree.

                                                                                                                                                In this specific case, Rust is not safer.

                                                                                                                                                I disagree here too.

                                                                                                                                                In this specific case, you need that perfect test suite. In either language.

                                                                                                                                                I partially agree. There is no such thing as a perfect test suite. A good crypto implementation should have a comprehensive test suite, of course, no matter the language. But that still isn’t as good as preventing these classes of bugs at compile time.

                                                                                                                                                Rust wouldn’t have saved me.

                                                                                                                                                Not really the point. Regardless of how lucky or skilled you are that there are no known critical vulnerabilities in these categories in your code, that disregards both unknown vulnerabilities in your code, and vulnerabilities in other people’s code as well. A safe language catches all three and scales; your method catches only one and doesn’t scale.

                                                                                                                                                1. 1

                                                                                                                                                  Note that I did go the extra mile and went a bit further than Valgrind & the sanitisers. I also happen to run Monocypher’s test suite under the TIS interpreter, and more recently TIS-CI (from TrustInSoft). Those things guarantee that they’ll catch any and all undefined behaviour, and they found a couple bugs the sanitisers didn’t.

                                                                                                                                                  that disregards both unknown vulnerabilities in your code

                                                                                                                                                  After that level of testing and a successful third party audit, I am confident there are none left.

                                                                                                                                                  and vulnerabilities in other people’s code as well

                                                                                                                                                  There is no such code. I have zero dependencies. Not even the standard library. The only thing I have to fear now is a compiler bug.

                                                                                                                                                  your method catches only one and doesn’t scale.

                                                                                                                                                  I went out of my way not to scale. Yet another peculiarity of modern cryptographic code, is that I don’t have to scale.

                                                                                                                                                  1. 1

                                                                                                                                                    There is no such code.

                                                                                                                                                    Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project? Because it seemed like a broader statement originally.

                                                                                                                                                    I went out of my way not to scale.

                                                                                                                                                    I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.

                                                                                                                                                    1. 1

                                                                                                                                                      Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project

                                                                                                                                                      I was talking about Monocypher specifically. Other projects do have dependencies, and any project that would use Monocypher almost certainly has dependencies, starting with system calls.

                                                                                                                                                      I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.

                                                                                                                                                      Fair enough. I was thinking from the project’s point of view: a given project only need one crypto library. A greenfield project can ditch backward compatibility and use a modern crypto library, which can be very small (or formally verified).

                                                                                                                                                      Yes, other people write cryptographic code. I myself added my own to this ever growing pile because I was unsatisfied with what we had (not even Libsodium was enough for me: too big, not easy to deploy). And the number of bugs in Monocypher + Libsodium is certainly higher than the number of bugs in Libsodium alone. No doubt about that.

                                                                                                                                                      Another reason why crypto libraries written in unsafe languages don’t scale, is the reputation game: it doesn’t matter how rigorously tested or verified my library is, if you don’t know it. And know it you cannot, unless you’re more knowledgeable than I am, and bother to audit my work yourself, which is prohibitively expensive. So in practice, you have to fall back to reputation and external signs: what other people say, the state of documentation, the security track record, issues from the bug tracker…

                                                                                                                                      2. 7

                                                                                                                                        This made me twitch!

                                                                                                                                        Why make a choice which prevents an entire class of bugs when you could simply put in extra time and effort to make sure you catch and fix them all?

                                                                                                                                        Why lock your doors when you can simply stand guard in front of them all night with a baseball bat?

                                                                                                                                        While personally would back the cryptography devs’ decision here, I think there is a legitimate discussion to be had around whether breaking compatibility for some long-standing users is the right thing to do. This post isn’t contributing well to that discussion.

                                                                                                                                      1. 3

                                                                                                                                        This is an old brain-dump, but I’m posting it in response to https://lobste.rs/s/mxcmxg/abstraction_is_okay_magic_is_not. I should definitely update this, and add whatever magic is. I don’t think “magic” is precisely the same as either of my three categories.

                                                                                                                                        1. 1

                                                                                                                                          It’s a good brain-dump, and I agree with it. I will note that my post (I wrote the ‘Abstraction is Okay’ article) was not exactly aimed at defining what abstraction is. Instead it’s just a musing on a pattern I’ve experienced as a consumer many times and how I consciously avoid producing it myself.

                                                                                                                                          I have concrete examples of “magic,” but I’d rather not share them because they’d tend to be very recognizable to the people who wrote them. Here’s an anonymized example I debated including in the post:

                                                                                                                                          @Wire(WidgetModel)
                                                                                                                                          @NormalPaths(exclude = AsyncPaths)
                                                                                                                                          @VarySerializer(determinant = ClientAuthentication)
                                                                                                                                          class WidgetService extends BasicModelService {
                                                                                                                                              // No code necessary!
                                                                                                                                          }
                                                                                                                                          

                                                                                                                                          What does this do? What does it accept? How do you modify it? It’s the kind of thing that someone presents and says proudly: “Look at how easy it is to implement a service!” but its name quickly becomes a portent of dread the minute they move on to another project or another job.

                                                                                                                                          “Hey, Beth, I was wondering how this thing in MagicalWeb works…”

                                                                                                                                          “Oh, man. Don’t worry about that crap, that’s Chuck’s stuff. He’s been gone for ages, and we’re trying to kill it. Add your path to the if statement in SkipThatCrap.java.”

                                                                                                                                        1. 21

                                                                                                                                          The article never mentions the, in my humble opinion, most important part of good logging practices and that is structured logging. Without it you end up with weird regexes or other hacks trying to parse your log messages.

                                                                                                                                          1. 4

                                                                                                                                            As a sibling post notes, if you use structured logging you’re mostly throwing away the idea that the entries must be easily parsable by a human. If that’s the case, and we’ll need a custom method of displaying the structured logs in a human friendly way, I believe we should forego plain text all together and gain the benefits of logging directly to binary.

                                                                                                                                            1. 5

                                                                                                                                              You can do human readable structured logging if you use key="value" formats inside text messages. Some people still prefer json, but there is a middle ground.

                                                                                                                                              1. 2

                                                                                                                                                If you need just key=value, that’s not really structured in my opinion.

                                                                                                                                                1. 4

                                                                                                                                                  Why not?

                                                                                                                                                  1. 2

                                                                                                                                                    Because the amount of information added by this format would be infinitesimal over a line based logger with manual tokenization. The reason why you’d want a structured logger is to allow proper context to a message. Unless you’re working with simple cases, the structure that would offer such context is more than one level deep.

                                                                                                                                                    1. 3

                                                                                                                                                      Hmm, definitely not.

                                                                                                                                                      Structured logging is about decorating log events with just enough of a schema to make them machine parseable, so that searching, aggregating, filtering, etc. can more than a crapshoot. Deeply nested events significantly increase the complexity of that schema, and therefore the requirements of the consumer.

                                                                                                                                                      By default, structured logs should be flat key/value pairs. It gets you the benefits of richer parseability, without giving up the ability to grep.

                                                                                                                                            2. 2

                                                                                                                                              Excellent point. That’s become such second nature to me by now, that I forgot to even mention it!

                                                                                                                                              1. 2

                                                                                                                                                I’m surprised it wasn’t mentioned, but the larger advantage of passing a logger around to constructors is the ability to then have nested named loggers, such as

                                                                                                                                                Battery.ChargingStatus.FileReader: Failed to open file { file: "/tmp/battery charge", error: ... }
                                                                                                                                                Battery.ChargingStatus: Failed to access status logs, skipping report
                                                                                                                                                
                                                                                                                                                1. 1

                                                                                                                                                  On top of that, structured logger if implemented properly, can often be faster and be operated at granular levels (like the other comments pointed out, sometimes you do want to on-fly turn on some logs at some locations, not all logs at all locations).

                                                                                                                                                  1. 1

                                                                                                                                                    I love structured logging, with one caveat: the raw messages emitted (let’s assume JSON) are harder for me to scan when tailing directly (which I usually only do locally as we have better log querying tools in the cloud), in contrast to a semi-structured simple key-value format. Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                                                                                                                                    1. 5

                                                                                                                                                      At Joyent, we used the Bunyan format. Each line in the file was a separate JSON object with standard properties, some mandatory and some optional, and freeform additional properties. We shipped a tool, bunyan, that was capable of acting as a filter that would render different human readable views of the JSON. For example, you would often run something like:

                                                                                                                                                      tail -F $(svcs -L manatee) | bunyan -o short
                                                                                                                                                      

                                                                                                                                                      It also had some rudimentary filtering options. It also had a relatively novel mode that would, instead of reading from a file or standard input, use DTrace probes for different log levels to allow you to dynamically listen for DEBUG and TRACE events even when those were not ordinarily present in the log files. The DTrace mode could target a particular process, or even all processes on the system that emitted Bunyan logs.

                                                                                                                                                      1. 1

                                                                                                                                                        Hi, what were the required fields? Was it just a unique request ID? Thanks for sharing about bunyan. Even though it’s been out for a while I was unaware of it.

                                                                                                                                                      2. 5

                                                                                                                                                        Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                                                                                                                                        We use JSON and the only tools I use are grep and jq. And although I am pretty much still a novice with these two, I found that with the power of shell piping I can do almost anything I want. Sometimes I reach for the Kibana web interface, get seriously confused and then go back to the command line to figure out how to do it there.

                                                                                                                                                        I wrote a simple tutorial for the process, just a couple of weeks ago.

                                                                                                                                                        1. 1

                                                                                                                                                          If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager? I feel like the systemd folks had the right idea even when everyone was making fun of them.

                                                                                                                                                          1. 3

                                                                                                                                                            I don’t think the average employer would be happy subsidizing an employee writing a log pager instead of implementing something that would bring a tangible result to the business. The potential money savings by using binary logs probably doesn’t outweigh the new subs/increased profits of churning out more features.

                                                                                                                                                            1. 1

                                                                                                                                                              To me that sounds like an excuse. The world is not made up of only software that is beholden to the all mighty shareholder.

                                                                                                                                                              1. 1

                                                                                                                                                                I mean, yes, if you’re developing something in your personal time, go bananas on what you implement.

                                                                                                                                                                But I also know my manager would look at me funny and ask why I’m not just shoving everything into CloudWatch/<cloud logging service>

                                                                                                                                                            2. 2

                                                                                                                                                              I’m sure most problems with systemd journals are fixable, but they’ve left a very bad taste in my mouth for two main reasons: if stuff gets deleted from under them they apparently never recover (my services continue to say something like “journal was rotated” until I restart them), and inspecting journals is incredibly slow. I’m talking magnitudes slower than log files. This is at its worst (I often have time to make a cup of tea) when piping the output into grep or, as journalctl already does by default, less, which means every byte has to be formatted by journalctl and copied only to be skipped over by its recipient. But it’s still pretty bad (I have time to complain on IRC about the wait) when giving journalctl filters that reduce the final output down to a few thousand lines, which makes me suspect that there are other less fundamental issues.

                                                                                                                                                              I should note that I’m using spinning disks and the logs I’m talking about are tens to hundreds of GB over a few months. I feel like that situation’s not abnormal.

                                                                                                                                                              1. 1

                                                                                                                                                                If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager?

                                                                                                                                                                It’s hard to imagine a case at work where I could justify writing my own log pager.
                                                                                                                                                                Here are some of the reasons I would avoid doing so:

                                                                                                                                                                • Logs are an incidental detail to the application.
                                                                                                                                                                • Logs are well understood; I can apply a logging library without issues.
                                                                                                                                                                • My application isn’t a beautiful and unique snowflake. I should use the same logging mechanisms and libraries as our other applications unless I can justify doing something different.
                                                                                                                                                                • JSON is boring, has a specification, substantial library support, tooling, etc.
                                                                                                                                                                • Specifying, documenting, and testing a custom format is a lot of work.
                                                                                                                                                                • Engineering time is limited; I try to focus my efforts on tasks that only I can complete.
                                                                                                                                                                1. 2

                                                                                                                                                                  Logs are an incidental detail to the application.

                                                                                                                                                                  I think this is trivially disproved by observing that if the logs stop working for your service, that is (hopefully!) a page-able event.

                                                                                                                                                                  Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                                                                                                                                  1. 1

                                                                                                                                                                    Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                                                                                                                                    I rely heavily on logging for the services I support but the applications I wrote for work have only error reporting. They are used by a small audience and problems are rare; I might get a crash report every 18 months or so.

                                                                                                                                                                    1. 1

                                                                                                                                                                      Ah, yeah, I presume the context here is services.

                                                                                                                                                              2. 1

                                                                                                                                                                Agreed. jq is a really nice tool. It made the decision to transition to using JSON for logging very easy.

                                                                                                                                                              3. 3

                                                                                                                                                                Don’t use JSON, use logfmt.

                                                                                                                                                                1. 1

                                                                                                                                                                  Yes! Logfmt is the good stuff. But it’s only semi-structured. Why not use JSON and a tool to transform to logfmt (with nested data elided probably) when needing to scan as a human?

                                                                                                                                                                  1. 1

                                                                                                                                                                    Logfmt is fully structured, it just doesn’t support nesting, which is an important feature! Structured logs should be flat.

                                                                                                                                                            1. 19

                                                                                                                                                              There’s a lot to agree with in this article, but why place the boundary between precompiled code and libraries downloaded in source form?

                                                                                                                                                              Do npm packages count as “editable code underneath that I have to read or write for these symbols to work”? Sure, if you’re downloading code and compiling it, you may have to edit it. It’s a risk. If you’re just hotlinking to a dll or something, that’s different. If you’re using precompiled code, whatever you’ve got, you’ve got.

                                                                                                                                                              Precompiled libraries have bugs too, may need workarounds, and can even be patched if you’re serious about it. I’d be more inclined to place the boundary between “code that is so well-tested that you probably don’t need to debug that code specifically” (which may include e.g. SQLite in source-code form, or very well-tested internal libraries) and “all other code” (which includes most commercial code.)

                                                                                                                                                              … but you have seen / considered a lot more software development than I did; if you have time, I’d be interested in seeing a bit more about why the “precompiled” vs. “source” distinction is the right place to draw the boundary?

                                                                                                                                                              1. 3

                                                                                                                                                                Yeah, this distinction makes no sense to me either. Maybe he’s assuming that anything worthy of being in a DLL musts be super solid and well tested? But that would be pretty naive.

                                                                                                                                                                1. 3

                                                                                                                                                                  I thought it was excluded because you can’t go and read through the source to know what it’s doing. So the risks to your product are still there, but there’s no additional cognitive load because you’ve just gotta take it on faith.

                                                                                                                                                                  1. 7

                                                                                                                                                                    If you took this seriously, open source/free software might be one of the worst things to ever happen to software development. We’d be obliged to shove everything into opaque blobs to have any hope.

                                                                                                                                                                    I assume the author meant something different, but it’s not quite clear.

                                                                                                                                                                    1. 5

                                                                                                                                                                      As long as that .DLL isn’t obfuscated. .NET DLLs written in C# that aren’t obfuscated are trivial to decompile between ILSpy and dotPeek, for example.

                                                                                                                                                                1. 1

                                                                                                                                                                  A really mixed bag, but #3 has never made sense to me, and has never stopped giving me problems, I just can’t train myself to not use select aliases in a where clause–I get it wrong, then fix the problem.

                                                                                                                                                                  1. 5

                                                                                                                                                                    Defenders of dynamically typed languages sometimes counter that these pitfalls do not matter when runtime failures are mostly harmless. If you want to find errors in your program, just run the program!

                                                                                                                                                                    … We do?

                                                                                                                                                                    1. 2

                                                                                                                                                                      Gabriel qualified the statement (“sometimes”) and gives an example (Nix). Is it not a reasonable accusation? What else can one do with a dynamically typed program, except running it?

                                                                                                                                                                      1. 2

                                                                                                                                                                        An non exhaustive list:

                                                                                                                                                                        • unit tests against your code
                                                                                                                                                                        • property tests against your code
                                                                                                                                                                        • runtime type pattern matching
                                                                                                                                                                        • runtime type conditionals
                                                                                                                                                                        • contracts
                                                                                                                                                                        • other compile-time checks besides a type system
                                                                                                                                                                        • defensive coding (for instance, calling int(x) BEFORE opening a file handle, not after)

                                                                                                                                                                        I love type systems, and even more so I love compile time guarantees, but any static typing advocate should be able to speak about the value of typing as a trade-off against other techniques and paradigms. On the other hand, no advocate of dynamic typing would (or should, I suppose) claim that, in the absence of the tangible and often productivity-enhancing benefits of strong static typing, “just run the program” is an acceptable alternative.

                                                                                                                                                                        1. 2

                                                                                                                                                                          An non exhaustive list:

                                                                                                                                                                          unit tests against your code property tests against your code runtime type pattern matching runtime type conditionals contracts other compile-time checks besides a type system defensive coding (for instance, calling int(x) BEFORE opening a file handle, not after)

                                                                                                                                                                          All but one of these is a variant on running the code.

                                                                                                                                                                          ** EDIT ** Well, the formatting didn’t stay and I’m on mobile so I’m just going to leave this as is for now…

                                                                                                                                                                          1. 4

                                                                                                                                                                            All but one of these is a variant on running the code.

                                                                                                                                                                            In that sense, one can say that even compilation is just a variant on running the code.

                                                                                                                                                                            1. 2

                                                                                                                                                                              I disagree. The compiler, a linter, or another static analyzer is another program operating on your program.

                                                                                                                                                                              Though, I suppose you could argue the same about unit tests and property tests. I think the difference is that they exercise your code whereas your code may not execute with a compiler. Things get a bit weird with compile time execution/macros/whatever.

                                                                                                                                                                              I don’t feel completely satisfied with my response.

                                                                                                                                                                              1. 6

                                                                                                                                                                                A typed program is two programs at once; the type system’s annotations themselves form a second program which is intertwined with the original program’s shape. This second program is the one which is run by the type-checker, and this is what gives us Rice’s Theorem for type systems, where e.g. GHC has some extensions that allow for type-checking to be undecideable.

                                                                                                                                                                                This also justifies why static type systems cannot possibly replace runtime testing of some sort; in order to complete in a reasonable time, the typical type system must be very simple, because its expressions are evaluated by the type-checker.

                                                                                                                                                                                I waffled over whether to post this at the top or the bottom of the thread, since I feel that this fact is at odds with how the discussion progressed. After all, what else can one do with a statically-typed program (that is, a pair of interleaved programs where one is not Turing-complete and is either sound or conservative over the second program). except running it or running just the type-checker portion? Similarly, it is not just obvious that compilation is a variant on running the code, in that is is an evaluation of one entire layer of the program’s code which produces a residual single-layer program, but also that compilation must run the code.

                                                                                                                                                                                1. 2

                                                                                                                                                                                  A typed program is two programs at once; the type system’s annotations themselves form a second program which is intertwined with the original program’s shape. This second program is the one which is run by the type-checker…

                                                                                                                                                                                  Okay, I am reasonably convinced by this. My major hang ups might have been implementation details: the distinction between your program and a program running your program can be blurry in some environments (e.g., Dr. Racket).

                                                                                                                                                                                  Given that there are two interleaved programs that may be run, the typed program’s advantage is that it is side effect free by it’s nature. Unit tests, property tests, fuzzers, etc. are running arbitrary code and could trigger arbitrary effects. I think that is ultimately the distinction that matters: how protected are you from mistakes in your program?

                                                                                                                                                                                  (Side note: running property tests on a function that sends email could be hilariously bad.)

                                                                                                                                                                                  I waffled over whether to post this at the top or the bottom of the thread, since I feel that this fact is at odds with how the discussion progressed.

                                                                                                                                                                                  Well, I appreciate being called out on my position.

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    On this view, what makes the type annotations the second and final program? Could it be three+ programs, since the linter/documentation generator also operate on a language that is intertwined with your program? This is a serious question, btw. I feel like I can think of some answers, but I don’t know that I can reason through them precisely.

                                                                                                                                                                                    1. 2

                                                                                                                                                                                      You’re quite right. There’s the view that syntax and semantics are the only two components to consider, and it’s a compelling view, rooted in the typical framing of Turing’s and Rice’s Theorems. There’s also the view that dependently-typed theories are at the ceiling of expressive power, and that our separation of types and values is just a happenstance due to choosing weak type systems, which also is tantalizing because of the universal nature of cubical and opetopic type theories. And to answer your point directly, there’s nothing stopping one syntax from having multiple different kinds of annotations which are projected in different ways.

                                                                                                                                                                                      I suppose I’m obligated to go with a category-oriented answer. Following Lawvere, syntax and semantics are adjoint and indeed form a Galois connection when viewed in a certain way. So, when we use Rice’s framing to split the properties of a program along such a connection, what we get is a pair of adjoint functors:

                                                                                                                                                                                      • On the left, semantics is a functor from programs to families of computer states
                                                                                                                                                                                      • On the right, syntax is a functor from families of computer states to programs
                                                                                                                                                                                      • The left side of the adjunction: the semantics of a program can include some particular behaviors, if and only if…
                                                                                                                                                                                      • The right side of the adjunction: those behaviors are encoded within the syntax of that program

                                                                                                                                                                                      And this is a family of adjunctions, with many possibilies for both functors. Rice’s Theorem says that we can’t decide whether most of the left-hand functors are computable, but the right-hand functor is not so encumbered; we can compute some properties of syntax, like well-typedness.

                                                                                                                                                                                      We can use the free-forgetful paradigm for analysis too: Semantics freely takes any of the possible execution paths, and syntax forgets which path was taken.

                                                                                                                                                                                      There is another functor which is left adjoint to semantics, the structure functor (discussed further here).

                                                                                                                                                                                      • On the left, structure is a functor from families of computer states to programs
                                                                                                                                                                                      • On the right, the same semantics as before
                                                                                                                                                                                      • The left side of the adjunction: the structure of some computations can all be found in a particular program, if and only if…
                                                                                                                                                                                      • The right side of the adjunction: the semantics of that program can reach all of those computations

                                                                                                                                                                                      In the free-forgetful paradigm, structure freely generates the program which it embodies, and semantics forgets structure.

                                                                                                                                                                                      Rice didn’t forbid us from analyzing this structure functor either! As a result, we have two different ways to examine a program without running it and invoking the dreaded semantics functor:

                                                                                                                                                                                      • We can study syntax backwards: Which machine states are possibly implied by our program? We can logically simplify the program text if it would help.
                                                                                                                                                                                      • We can study structure backwards: Which machine states do our program require or assume? We can simplify the states if it would help.

                                                                                                                                                                                      This means that analyzing structure is abstract interpretation! Or I’ve horribly misunderstood something.

                                                                                                                                                                                      Now I’m quite sorry to have not posted this at the top of the thread! Oh well.

                                                                                                                                                                                    2. 1

                                                                                                                                                                                      How does this interpretation handle fully inferred types? Is that not truly an analysis pass on the program, I.e. running a program on the source code, not running the source code itself?

                                                                                                                                                                                      I’m not sure if there’s any value in making the distinction… But it feels like there must be, because static types are qualitatively different from dynamic types.

                                                                                                                                                                                      1. 1

                                                                                                                                                                                        The inference is usually syntactic and doesn’t require examining the semantics of the program. This is the sense in which the second program is intertwined with the first program.

                                                                                                                                                                                        For example, in many languages, 42 has a type like Int. This fact could be known by parsers. Similarly, (\x -> x + 1) has a type like Int -> Int. Parsers could know this too, since function types are built out of other types. In general, syntactic analysis can discover both the concrete parse tree and also the inferred type.

                                                                                                                                                                                  2. 2

                                                                                                                                                                                    Types categorise values, they don’t run code.