1. 4

    You can filter out the release tag if you don’t want to see these posts.

    1. 13

      Wait, what? Lobste.rs have a tag for merkle-trees?

      Really nice writeup, thanks!

      1. 19

        At one point it was called cryptocurrencies but it got renamed after one too many nontechnical cryptocurrency story got posted.

      1. 5

        Meanwhile, PyPy is around 4x faster than CPython.

        1. 6

          Annecdote ain’t data, but I’ve never been successful at getting PyPy to provide improved performance. My use cases have been things like running tooling (Pylint is extremely slow under PyPy, much moreso than CPython), just running web apps, and a lot of other things that aren’t benchmarks.

          I don’t want to be too critical of PyPy, I imagine it gets a lot of what a lot of people want. But I don’t know what real workloads end up benefiting from it.

          1. 4

            PyPy upstream generally treats slowness as a bug and is willing to expend resources to fix it, if you’re willing to file issues with minimal test cases. (Here is a recent example bug about slowness.)

            Anecdotes aren’t data, but about a decade ago, I ported a Minecraft server from Numpy and CPython to array.array and PyPy, and at the time, I recorded a 60x speedup on a microbenchmark, and around a 20x speedup for typical gameplay interactions, resulting in a backend that spent most of its time sleeping and waiting for I/O.

            As long as we’re on the topic, it’s worth knowing that PyPy comes with a toolkit, RPython, which allows folks to generate their own JITs from Python. So, if one wanted more speed than was available with Python’s language design, then PyPy provides a route for forking the interpreter and standard library, and making arbitrarily distant departures from Python while still having high performance. For example, if we can agree that Dolphin implements “real workloads”, then PyGirl (code, paper) probably does as well.

            1. 3

              Yeah to me it helps to think of workloads in these categories (even if there are obviously way more than this, and way more dimensions)

              1. String / hash / object workloads (similar to web apps. Similar to a linter, and similar to Oil’s parser)
              2. Numeric workloads (what people write Cython extensions for; note that NumPy is written largely in Cython.)

              JITs are a lot better at the second type of workload than the first. My experience matches yours – when I tried running Oil with PyPy, it was slower and used more memory, not faster.

              Also, I think that workload 1 is the more important one for Python. If I want to write fast numeric code, it’s not painful to do in C++. On the other hand, doing string/hash/object graph workloads in C++ is very painful. It’s also less-than-great in Rust, particularly graphs.

              So while I think PyPy is an astonishing project (and that impression grows after learning more about how it works), I also think it doesn’t speed up the most important workloads in Python. Not that I think any other effort will do so – the problems are pretty fundamental and there have been a couple decades of attempts.

              (In contrast I got much better performance results adding static types manually, and semi-automatically translating Oil to C++. This is not a general solution as its labor intensive and restricts the language, although there are some other benefits to that.)

              1. 1

                I see the outline of your point, but I’m not sure on the specifics. In particular, a mechanism is missing: What makes strings, dictionaries, and user-customized classes inherently hard to JIT, particularly with a PyPy-style tracing metainterpreter?

                Edit: Discussion in #pypy on Freenode yielded the insight that CPUs have trouble with anything which is not in their own list of primitive types, requiring composite operations for composite types. Since JITs compile to CPU instructions, they must struggle with instruction selection for composite types. A lesson for language designers is to look for opportunities to provide new primitive object implementations, using the CPU’s existing types in novel ways.

                Our experience in the Monte world is that our RPython-generated JIT successfully speeds up workloads like parsing and compiling Monte modules to bytecode, a task which is string- and map-heavy. Our string and map objects are immutable, and this helps the JIT remove work.

                1. 1

                  Yes the JITs do a lot better on integers and floats because they’re machine types.

                  The performance of strings and hash tables is sort of “one level up”, and the JITs don’t seem to help much at that level (and for some reason lots of people seem to misunderstand this.)

                  As an anecdote, when Go was released, there were some benchmarks where it was slower than Python, just because Python’s hash tables were more optimized. And obviously Go is compiled and Python is interpreted, but that was still true. So that is a similar issue.

                  So there are many dimensions to performance, and many workloads. Saying “4x faster” is doing violence to reality. In some cases it’s the difference between being able to use PyPy and not being able to use it.

                2. 1

                  SciPy has some cython code along with a bunch of fortran code but NumPy is all C.

                  1. 1

                    Ah sorry you are right, I think I was remembering Pandas, which has a lot of Cython in its core:

                    https://github.com/pandas-dev/pandas/tree/master/pandas/_libs

                  2. 1

                    cython is also a translator to C. why didn’t you use cython for oil?

                    1. 1

                      It generates code that depends on the Python runtime, and Cython is a different language than statically-typed Python. I don’t want to be locked into the former, and translating the code is probably even more labor intensive than what I’m doing (I leveraged MyPy team work on automatic type annotation etc.). It also wouldn’t be fast enough as far as I can tell.

                  3. 3

                    pypy is 4x faster…. for long-running tasks that allow the jit to warm up. Lots of python workloads (e.g. pylint) run the interpreter as a one-off so pypy won’t help there. Interpreter startup speed is also critical for one-off workflows and pypy isn’t optimized for that either.

                    1. 3

                      I think it’s more like 10x-100x faster OR 10% slower for different workloads – “4x” doesn’t really capture it. See my sibling comment about string/hash/object vs. numeric workloads.

                    2. 2

                      I used PyPy recently, for the first time and I had a nice experience. I am experimenting with SQLite and trying to figure out the fast ways to insert 1B rows. My CPython version was able to insert 100M rows in 500 is seconds, same in PyPy took 150 seconds.

                      The best part was, I did not have to change anything in my original code. It was just drop in, as advertised. Ran it with PyPy and got the speed bumps.

                    3. 2

                      Specifically, we want to achieve these performance goals with CPython to benefit all users of Python including those unable to use PyPy or other alternative virtual machines.

                      1. 1

                        Apparently the goal is a 2x speed up by 3.11 and a 5x speed up in 4 years.

                        1. 4

                          Yes. Assuming that those numbers are not exaggerated, I expect that PyPy will still be faster than CPython year after year. The reasoning is due to the underlying principle that most improvements to CPython can be ported to PyPy since they have similar internal structure.

                          In GvR’s slides, they say that they “can’t change base layout, object layout”. This is the only part of PyPy’s interpreter which is structurally different from CPython. The same slide lists components which PyPy derived directly from CPython: the bytecode, the stack frames, the bytecode compiler, and bytecode interpreter.

                          Specializing bytecode has been tried for Python before; I recall a paper which monomorphized integers and other common builtin types. These approaches tend to fail unless they can remove some interpretative overhead. I expect that a more useful product of this effort will be a better memory model and simpler bytecodes, rather than Shannon’s grand explosion of possible bytecode arrangements.

                          1. 1

                            I’m curious about mypyc personally. Seems to me like (c)python is just hard to optimize and depends too much on implementation details (the C API) to be changed; to get a significant leap in performance it seems like using a statically typed, less dynamic subset, would give significantly higher speedups. Of course the downside is that it doesn’t work for old code (unless it happens to be in this fragment).

                            1. 1

                              Monomorphizing code does not always speed it up. There are times when tags/types can be checked for free, thanks to the dominating effects of cache thrashing, and so the cost of dynamically-typed and statically-typed traversals ends up being similar.

                              It’s not an accident that some half-dozen attempts to monomorphize CPython internals have failed, while PyPy’s tracing JIT is generally effective. Monomorphization can remove inner-interpreter work, but not interpretative overhead.

                              1. 2

                                Well by “less dynamic” I also mean not having a dictionary per class and this kind of stuff :-). I should have been clearer. tag checks is one thing, but performing dictionary lookups all the time to resolve identifiers or fields is also very heavy. The statically typed aspect, I have no idea if it’s truly necessary, but it’d make it easier to implement, right?

                      1. 50

                        The paper has this to say (page 9):

                        Regarding potential human research concerns. This experiment studies issues with the patching process instead of individual behaviors, and we do not collect any personal information. We send the emails to the Linux community and seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.

                        [..]

                        Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.

                        I’m not familiar with the generally accepted standards on these kind of things, but this sounds rather iffy to me. I’m very far removed from academia, but I’ve participated in a few studies over the years, which were always just questionaries or interviews, and even for those I had to sign a consent waiver. “It’s not human research because we don’t collect personal information” seems a bit strange.

                        Especially since the wording “we will have to report this, AGAIN, to your university” implies that this isn’t the first time this has happened, and that the kernel folks have explicitly objected to being subject to this research before this patch.

                        And trying to pass off these patches as being done in good faith with words like “slander” is an even worse look.

                        1. 78

                          They are experimenting on humans, involving these people in their research without notice or consent. As someone who is familiar with the generally accepted standards on these kinds of things, it’s pretty clear-cut abuse.

                          1. 18

                            I would agree. Consent is absolutely essential but just one of many ethical concerns when doing research. I’ve seen simple usability studies be rejected due to lesser issues.

                            It’s pretty clear this is abuse.. the kernel team and maintainers feel strongly enough to ban the whole institution.

                            1. 10

                              Yeah, agreed. My guess is they misrepresented the research to the IRB.

                              1. 3

                                They are experimenting on humans

                                This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                                1. 19

                                  A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                                  1. 0

                                    A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                                    This statement is so vague as to be almost meaningless. In what relevant ways is a professional penetration testing contract (or, more relevantly, the associated process) different from this particular research project? Which of the two should be held to a higher standard? Why? What does “held to a higher standard” even mean?

                                    Moreover, that claim doesn’t actually have anything to do with the comment I was replying to, which was claiming that this project was “experimenting on humans”. It doesn’t matter whether or not something is “research” or “industry” for the purposes of whether or not it’s “human experimentation” - either it is, or it isn’t.

                                    1. 18

                                      Resident pentester and ex-academia sysadmin checking in. I totally agree with @Foxboron and their statement is not vague nor meaningless. Generally in a penetration test I am following basic NIST 800-115 guidance for scoping and target selection and then supplement contractual expectations for my clients. I can absolutely tell you that the methodologies that are used by academia should be held to a higher standard in pretty much every regard I could possibly come up with. A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data.

                                      Let’s put it in real terms, I am hired to do a security assessment in a very fixed highly focused set of targets explicitly defined in contract by my client in an extremely fixed time line (often very short… like 2 weeks maximum and 5 day average). Guess what happens if social engineering is not in my contract? I don’t do it.

                                      1. 1

                                        Resident pentester and ex-academia sysadmin checking in.

                                        Note: this is worded like an appeal to authority, although you probably don’t mean it that way, so I’m not going to act like you are.

                                        I totally agree with @Foxboron and their statement is not vague nor meaningless.

                                        Those are two completely separate things, and neither is implied by the other.

                                        their statement is not vague nor meaningless.

                                        Not true - their statement contained none of the information you just provided, nor any other sort of concrete or actionable information - the statement “hold to a higher standard” is both vague and meaningless by itself…and it was by itself in that comment (or, obviously, there were other words - none of them relevant) - there was no other information.

                                        the methodologies that are used by academia should be held to a higher standard

                                        Now you’re mixing definitions of “higher standard” - GP and I were talking about human experimentation and ethics, while you seem to be discussing rigorousness and reproducibility of experiments (although it’s not clear, because “A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data” is slightly ambiguous).

                                        None of the above is relevant to the question of “was this a human experiment” and the closely-related one “is penetration testing a human experiment”. Evidence suggests “no” given that the term does not appear in that document, nor have I heard of any pentest being reviewed by an ethics review board, nor have I heard any mention of “human experimenting” in the security community (including when gray-hat and black-hat hackers and associated social engineering e.g. Kevin Mitnick are mentioned), nor are other similar, closer-to-human experimentation (e.g. A/B testing, which is far closer to actually experimenting on people) processes considered to be such - up until this specific case.

                                      2. 5

                                        if you’re an employee in an industry, you’re either informed of penetration testing activity, or you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment.

                                        if a company did this to their employees without any warning, they’d be shitty too, but the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific is nonzero.

                                        either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                        1. 1

                                          you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment

                                          By this logic, you’ve also agreed to everything else in a massive, hundred-page long EULA that you click “I agree” on, as well as consent to be tracked by continuing to use a site that says that in a banner at the bottom, as well as consent to Google/companies using your data for whatever they want and/or selling it to whoever will buy.

                                          …and that’s ignoring whether or not companies that have pentesting done on them actually explicitly include that specific warning in your contract - “implicit” is not good enough, as then anyone can claim that, as a Linux kernel patch reviewer, you’re “implicitly agreeing that you may be exposed to the risk of social engineering for the purpose of getting bad code into the kernel”.

                                          the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific

                                          Like others, you’re mixing up the issue of whether the experiment was properly-designed with the issue of whether it was human experimentation. I’m not making any attempt to argue the former (because I know very little about how to do good science aside from “double-blind experiments yes, p-hacking no”), so I don’t know why you’re arguing against it in a reply to me.

                                          either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                          I completely agree that the goals are different - but again, that’s irrelevant for determining whether or not something is “human experimentation”. Doesn’t matter what the motive is, experimenting on humans is experimenting on humans.

                                    2. 18

                                      This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                                      I had a former colleague that once bragged about getting someone fired at his previous job during a pentesting exercise. He basically walked over to this frustrated employee at a bar, bribed him a ton of money and gave a job offer in return for plugging a usb key into the network. He then reported it to senior management and the employee was fired. While that is an effective demonstration of a vulnerability in their organization, what he did was unethical under many moral frameworks.

                                      1. 2

                                        First, the researchers didn’t engage in any behavior remotely like this.

                                        Second, while indeed an example of pentesting, most pentesting is not like this.

                                        Third, the fact that it was “unethical under many moral frameworks” is irrelevant to what I’m arguing, which is that the study was not “human experimentation”. You can steal money from someone, which is also “unethical under many moral frameworks”, and yet still not be doing “human experimentation”.

                                      2. 3

                                        If there is a pentest contract, then there is consent, because consent is one of the pillars of contract law.

                                        1. 1

                                          That’s not an argument that pentesting is human experimentation in the first place.

                                    3. 42

                                      The statement from the UMinn IRB is in line with what I heard from the IRB at the University of Chicago after they experimented on me, who said:

                                      I asked about their use of any interactions, or use of information about any individuals, and they indicated that they have not and do not use any of the data from such reporting exchanges other than tallying (just reports in aggregate of total right vs. number wrong for any answers received through the public reporting–they said that much of the time there is no response as it is a public reporting system with no expectation of response) as they are not interested in studying responses, they just want to see if their tool works and then also provide feedback that they hope is helpful to developers. We also discussed that they have some future studies planned to specifically study individuals themselves, rather than the factual workings of a tool, that have or will have formal review.

                                      They because claim they’re studying the tool, it’s OK to secretly experiment on random strangers without disclosure. Somehow I doubt they test new drugs by secretly dosing people and observing their reactions, but UChicago’s IRB was 100% OK with doing so to programmers. I don’t think these IRBs literally consider programmers sub-human, but it would be very inconvenient to accept that experimenting on strangers is inappropriate, so they only want to do so in places they’ve been forced to by historical abuse. I’d guess this will continue for years until some random person is very seriously harmed by being experimented on (loss of job/schooling, pushing someone unstable into self-harm, targeting someone famous outside of programming) and then over the next decade IRBs will start taking it seriously.

                                      One other approach that occurs to me is that the experimenters and IRBs claim they’re not experimenting on their subjects. That’s obviously bullshit because the point of the experiment is to see how the people respond to the treatment, but if we accept the lie it leaves an open question: what is the role played by the unwitting subject? Our responses are tallied, quoted, and otherwise incorporated into the results in the papers. I’m not especially familiar with academic publishing norms, but perhaps this makes us unacknowledged co-authors. So maybe another route to stopping experimentation like this would be things like claiming copyright over the papers, asking journals for the papers to be retracted until we’re credited, or asking the universities to open academic misconduct investigations over the theft of our work. I really don’t have the spare attention for this, but if other subjects wanted to start the ball rolling I’d be happy to sign on.

                                      1. 23

                                        I can kind of see where they’re coming from. If I want to research if car mechanics can reliably detect some fault, then sending a prepared car to 50 garages is probably okay, or at least a lot less iffy. This kind of (informal) research is actually fairly commonly by consumer advocacy groups and the like. The difference is that the car mechanics will get paid for their work where as the Linux devs and you didn’t.

                                        I’m gonna guess the IRBs probably aren’t too familiar with the dynamics here, although the researchers definitely were and should have known better.

                                        1. 18

                                          Here it’s more like keying someone’s car to see how quick it takes them to get an insurance claim.

                                          1. 4

                                            Am I misreading? I thought the MR was a patch designed to fix a potential problem, and the issue was

                                            1. pushcx thought it wasn’t a good fix (making it a waste of time)
                                            2. they didn’t disclose that it was an auto-generated PR.

                                            Those are legitimate complaints, c.f. https://blog.regehr.org/archives/2037, but from the analogies employed (drugs, dehumanization, car-keying), I have to double-check that I haven’t missed an aspect of the interaction that makes it worse than it seemed to me.

                                            1. 2

                                              We were talking about Linux devs/maintainers too, I commented on that part.

                                              1. 1

                                                Gotcha. I missed that “here” was meant to refer to the Linux case, not the Lobsters case from the thread.

                                          2. 1

                                            Though there they are paying the mechanic.

                                          3. 18

                                            IRB is a regulatory board that is there to make sure that researchers follow the (Common Rule)[https://www.hhs.gov/ohrp/regulations-and-policy/regulations/common-rule/index.html].

                                            In general, any work that receives federal funding needs to comply with the federal guidelines for human subject research. All work involving human subjects (usually defined as research activities that involve interaction with humans) need to be reviewed and approved by the institution IRB. These approvals fall within a continuum, from a full IRB review (which involve the researcher going to a committee and explaining their work and usually includes continued annual reviews) to a declaration of the work being exempt from IRB supervision (usually this happens when the work meets one of the 7 exemptions listed in the federal guidelines). The whole process is a little bit more involved, see for example (all the charts)[https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts/index.html] to figure this out.

                                            These rules do not cover research that doesn’t involve humans, such as research on technology tools. I think that there is currently a grey area where a researcher can claim that they are studying a tool and not the people interacting with the tool. It’s a lame excuse that probably goes around the spirit of the regulations and is probably unethical from a research stand point. The data aggregation method or the data anonymization is usually a requirement for an exempt status and not a non-human research status.

                                            The response that you received from IRB is not surprising, as they probably shouldn’t have approved the study as non-human research but now they are just protecting the institution from further harm rather than protecting you as a human subject in the research (which, by the way, is not their goal at this point).

                                            One thing that sticks out to me about your experience is that you weren’t asked to give consent to participate in the research. That usually requires a full IRB review as informed consent is a requirement for (most) human subject research. Exempt research still needs informed consent unless it’s secondary data analysis of existing data (which your specific example doesn’t seem to be).

                                            One way to quickly fix it is to contact the grant officer that oversees the federal program that is funding the research. A nice email stating that you were coerced to participate in the research study by simply doing your work (i.e., review a patch submitted to a project that you lead) without being given the opportunity to provide prospective consent and without receiving compensation for your participation and that the research team/university is refusing to remove your data even after you contacted them because they claim that the research doesn’t involve human subjects can go a long way to force change and hit the researchers/university where they care the most.

                                            1. 7

                                              Thanks for explaining more of the context and norms, I appreciate the introduction. Do you know how to find the grant officer or funding program?

                                              1. 7

                                                It depends on how “stalky” you want to be.

                                                If NSF was the funder, they have a public search here: https://nsf.gov/awardsearch/

                                                Most PIs also add a line about grants received to their CVs. You should be able to match the grant title to the research project.

                                                If they have published a paper from that work, it should probably include an award number.

                                                Once you have the award number, you can search the funder website for it and you should find a page with the funding information that includes the program officer/manager contact information.

                                                1. 3

                                                  If they published a paper about it they likely included the grant ID number in the acknowledgements.

                                                  1. 1

                                                    You might have more luck reaching out to the sponsored programs office at their university, as opposed to first trying to contact an NSF program officer.

                                                2. 4

                                                  How about something like a an Computer Science - External Review Board? Open source projects could sign up, and include a disclaimer that their project and community ban all research that hasn’t been approved. The approval process could be as simple as a GitHub issue the researcher has to open, and anyone in the community could review it.

                                                  It wouldn’t stop the really bad actors, but any IRB would have to explain why they allowed an experiment on subjects that explicitly refused consent.

                                                  [Edit] I felt sufficiently motivated, so I made a quick repo for the project . Suggestions welcome.

                                                  1. 7

                                                    I’m in favor of building our own review boards. It seems like an important step in our profession taking its reponsibility seriously.

                                                    The single most important thing I’d say is, be sure to get the scope of the review right. I’ve looked into this before and one of the more important limitations on IRBs is that they aren’t allowed to consider the societal consequences of the research succeeding. They’re only allowed to consider harm to experimental subjects. My best guess is that it’s like that because that’s where activists in the 20th-century peace movement ran out of steam, but it’s a wild guess.

                                                    1. 4

                                                      At least in security, there are a lot of different Hacker Codes of Ethics floating around, which pen testers are generally expected to adhere to… I don’t think any of them cover this specific scenario though.

                                                      1. 2

                                                        any so-called “hacker code of ethics” in use by any for-profit entity places protection of that entity first and foremost before any other ethical consideration (including human rights) and would likely not apply in a research scenario.

                                                  2. 23

                                                    They are bending the rules for non human research. One of the exceptions for non-human research is research on organization, which my IRB defines as “Information gathering about organizations, including information about operations, budgets, etc. from organizational spokespersons or data sources. Does not include identifiable private information about individual members, employees, or staff of the organization.” Within this exception, you can talk with people about how the organization merges patches but not how they personally do that (for example). All the questions need to be about the organization and not the individual as part of the organization.

                                                    On the other hand, research involving human subjects is defined as any research activity that involves an “individual who is or becomes a participant in research, either:

                                                    • As a recipient of a test article (drug, biologic, or device); or
                                                    • As a control.”

                                                    So, this is how I interpret what they did.

                                                    The researchers submitted an IRB approval saying that they just downloaded the kernel maintainer mailing lists and analyzed the review process. This doesn’t meet the requirements for IRB supervision because it’s either (1) secondary data analysis using publicly available data and (2) research on organizational practices of the OSS community after all identifiable information is removed.

                                                    Once they started emailing the list with bogus patches (as the maintainers allege), the research involved human subjects as these people received a test article (in the form of an email) and the researchers interacted with them during the review process. The maintainers processing the patch did not do so to provide information about their organization’s processes and did so in their own personal capacity (In other words, they didn’t ask them how does the OSS community processes this patch but asked them to process a patch themselves). The participants should have given consent to participate in the research and the risks of participating in it should have been disclosed, especially given the fact that missing a security bug and agreeing to merge it could be detrimental to someone’s reputation and future employability (that is, this would qualify for more than minimal risk for participants, requiring a full IRB review of the research design and process) with minimal benefits to them personally or to the organization as a whole (as it seems from the maintainers’ reaction to a new patch submission).

                                                    One way to design this experiment ethically would have been to email the maintainers and invite them to participate in a “lab based” patch review process where the research team would present them with “good” and “bad” patches and ask them whether they would have accepted them or not. This is after they were informed about the study and exercised their right to informed consent. I really don’t see how emailing random stuff out and see how people interact with it (with their full name attached to it and in full view of their peers and employers) can qualify as research with less than minimal risks and that doesn’t involve human subjects.

                                                    The other thing that rubs me the wrong way is that they sought (and supposedly received) retroactive IRB approval for this work. That wouldn’t fly with my IRB, as my IRB person would definitely rip me a new one for seeking retroactive IRB approval for work that is already done, data that was already collected, and a paper that is already written and submitted to a conference.

                                                    1. 6

                                                      You make excellent points.

                                                      1. IRB review has to happen before the study is started. For NIH, the grant application has to have the IRB approval - even before a single experiment is even funded to be done, let alone actually done.
                                                      2. I can see the value of doing a test “in the field” so as to get the natural state of the system. In a lab setting where the participants know they are being tested, various things will happen to skew results. The volunteer reviewers might be systematically different from the actual population of reviewers, the volunteers may be much more alert during the experiment and so on.

                                                      The issue with this study is that there was no serious thought given to what are the ethical ramifications of this are.

                                                      If the pen tested system has not asked to be pen tested then this is basically a criminal act. Otherwise all bank robbers could use the “I was just testing the security system” defense.

                                                      1. 8

                                                        The same requirement for prior IRB approval is necessary for NSF grants (which the authors seem to have received). By what they write in the paper and my interpretation of the circumstances, they self certified as conducting non-human research at time of submitting the grant and only asked their IRB for confirmation after they wrote the paper.

                                                        Totally agree with the importance of “field experiment” work and that, sometimes, it is not possible to get prospective consent to participate in the research activities. However, the guidelines are clear on what activities fall within research activities that are exempt from prior consent. The only one that I think is applicable to this case is exception 3(ii):

                                                        (ii) For the purpose of this provision, benign behavioral interventions are brief in duration, harmless, painless, not physically invasive, not likely to have a significant adverse lasting impact on the subjects, and the investigator has no reason to think the subjects will find the interventions offensive or embarrassing. Provided all such criteria are met, examples of such benign behavioral interventions would include having the subjects play an online game, having them solve puzzles under various noise conditions, or having them decide how to allocate a nominal amount of received cash between themselves and someone else.

                                                        These usually cover “simple” psychology experiments involving mini games or economics games involving money.

                                                        In the case of this kernel patching experiment, it is clear that this experiment doesn’t meet this requirement as participants have found this intervention offensive or embarrassing, to the point that they are banning the researchers’ institution from pushing patched to the kernel. Also, I am not sure if reviewing a patch is a “benign game” as this is the reviewers’ jobs, most likely. Plus, the patch review could have adverse lasting impact on the subject if they get asked to stop reviewing patches if they don’t catch the security risk (e.g., being deemed imcompetent).

                                                        Moreover, there is this follow up stipulation:

                                                        (iii) If the research involves deceiving the subjects regarding the nature or purposes of the research, this exemption is not applicable unless the subject authorizes the deception through a prospective agreement to participate in research in circumstances in which the subject is informed that he or she will be unaware of or misled regarding the nature or purposes of the research.

                                                        As their patch submission process was deceptive in nature, as their outline in the paper, exemption 3(ii) cannot apply to this work unless they notify maintainers that they will be participating in a deceptive research study about kernel patching.

                                                        That leaves the authors to either pursue full IRB review for their work (as a full IRB review can approve a deceptive research project if it deems it appropriate and the risk/benefit balance is in favor to the participants) or to self-certify as non-human subjects research and fix any problems later. They decided to go with the latter.

                                                    2. 35

                                                      We believe that an effective and immediate action would be to update the code of conduct of OSS, such as adding a term like “by submitting the patch, I agree to not intend to introduce bugs.”

                                                      I copied this from that paper. This is not research, anyone who writes a sentence like this with a straight face is a complete moron and is just mocking about. I hope all of this will be reported to their university.

                                                      1. 18

                                                        It’s not human research because we don’t collect personal information

                                                        I yelled bullshit so loud at this sentence that it woke up the neighbors’ dog.

                                                        1. 2

                                                          Yeah, that came from the “clarifiactions” which is garbage top to bottom. They should have apologized, accepted the consequences and left it at that. Here’s another thing they came up with in that PDF:

                                                          Suggestions to improving the patching process In the paper, we provide our suggestions to improve the patching process.

                                                          • OSS projects would be suggested to update the code of conduct, something like “By submitting the patch, I agree to not intend to introduce bugs”

                                                          i.e. people should say they won’t do exactly what we did.

                                                          They acted in bad faith, skirted IRB through incompetence (let’s assume incompetence and not malice) and then act surprised.

                                                        2. 14

                                                          Apparently they didn’t ask the IRB about the ethics of the research until the paper was already written: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf

                                                          Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. We apologize for the raised concerns. This is an important lesson we learned—Do not trust ourselves on determining human research; always refer to IRB whenever a study might be involving any human subjects in any form. We would like to thank the people who suggested us to talk to IRB after seeing the paper abstract.

                                                          1. 14

                                                            I don’t approve of researchers YOLOing IRB protocols, but I also want this research done. I’m sure many people here are cynical/realistic enough that the results of this study aren’t surprising. “Of course you can get malicious code in the kernel. What sweet summer child thought otherwise?” But the industry as a whole proceeds largely as if that’s not the case (or you could say that most actors have no ability to do anything about the problem). Heighten the contradictions!

                                                            There are some scary things in that thread. It sounds as if some of the malicious patches reached stable, which suggests that the author mostly failed by not being conservative enough in what they sent. Or for instance:

                                                            Right, my guess is that many maintainers failed in the trap when they saw respectful address @umn.edu together with commit message saying about “new static analyzer tool”.

                                                            1. 17

                                                              I agree, while this is totally unethical, it’s very important to know how good the review processes are. If one curious grad student at one university is trying it, you know every government intelligence department is trying it.

                                                              1. 8

                                                                I entirely agree that we need research on this topic. There’s better ways of doing it though. If there aren’t better ways of doing it, then it’s the researcher’s job to invent them.

                                                              2. 7

                                                                It sounds as if some of the malicious patches reached stable

                                                                Some patches from this University reached stable, but it’s not clear to me that those patches also introduced (intentional) vulnerabilities; the paper explicitly mentions the steps that they’re taking steps to ensure those patches don’t reach stable (I omitted that part, but it’s just before the part I cited)

                                                                All umn.edu are being reverted, but at this point it’s mostly a matter of “we don’t trust these patches and will need additional review” rather than “they introduced security vulnerabilities”. A number of patches already have replies from maintainers indicating they’re genuine and should not be reverted.

                                                                1. 5

                                                                  Yes, whether actual security holes reached stable or not is not completely clear to me (or apparently to maintainers!). I got that impression from the thread, but it’s a little hard to say.

                                                                  Since the supposed mechanism for keeping them from reaching stable is conscious effort on the part of the researchers to mitigate them, I think the point may still stand.

                                                                  1. 1

                                                                    It’s also hard to figure out what the case is since there is no clear answer what the commits where, and where they are.

                                                                2. 4

                                                                  The Linux review process is so slow that it’s really common for downstream folks to grab under-review patches and run with them. It’s therefore incredibly irresponsible to put patches that you know introduce security vulnerabilities into this form. Saying ‘oh, well, we were going to tell people before they were deployed’ is not an excuse and I’d expect it to be a pretty clear-cut violation of the Computer Misuse Act here and equivalent local laws elsewhere. That’s ignoring the fact that they were running experiments on people without their consent.

                                                                  I’m pretty appalled the Oakland accepted the paper for publication. I’ve seen paper rejected from there before because they didn’t have appropriate ethics review oversite.

                                                              1. 1

                                                                Flagged as spam, this is an ad. Also don’t put the site’s name in the story title.

                                                                1. 1

                                                                  Ah uff! Is there any context in which it is ok to share about our own work?

                                                                  1. 6

                                                                    When you’ve contributed things to the community in the past year besides “here’s my project” and “here’s my project again”

                                                                1. 12

                                                                  You can also join #lobsters-advent on freenode to chat about problems.

                                                                  1. 1

                                                                    Joined!

                                                                  1. 10

                                                                    Why does software specifically need to support Apple Silicon, not just Aarch64, to run natively? The instruction set is 9 years old, runs in every mobile device, and has been the second most important instruction set to support for the last 5 years or so: Everyone should expect support for this silicon from every programming language worth using by now.

                                                                    I’m very surprised that Fortran and Go somehow don’t support it already, and even that general software, no matter if it’s compiled through Homebrew, has issues being compiled on ARM. Such microscopic problems should evaporate pretty quickly once exposed, assuming all APIs stay the same.

                                                                    What I’m unsurprised about is that JIT runtimes and other software with heavy assembly optimization are more or less lacking ARM/NEON optimizations, because that takes human labor. Also relevant to future proofing, I would like to see a Dav1d benchmark. It should be one of the better optimized code bases by now.

                                                                    1. 10

                                                                      There is an aarch64 port of gfortran, it’s used for e.g. raspbian. However there isn’t yet a stable aarch64 port for darwin as there are substantial ABI differences compared with linux. See the tracking issue.

                                                                      1. 4

                                                                        Why does software specifically need to support Apple Silicon, not just Aarch64

                                                                        For most software x86 vs ARM doesn’t even matter and it is just a recompile. Most software doesn’t know or even care what architecture it runs on.

                                                                        Where it does matter, and get a lot more complicated, is with software that does interact with the CPU or the OS at a much lower level. Compilers, code generators, JITs, highly optimized code that uses assembly code.

                                                                        Brew mostly has issues because of build system issues.

                                                                        1. 2

                                                                          Why does software specifically need to support Apple Silicon, not just Aarch64, to run natively?

                                                                          This is me being cynical, but I expect Apple to start extending Aarch64 with custom instructions any day now. Have to wonder how ARM feels about that.

                                                                          1. 5

                                                                            They’ve been shipping their own ARM chips for a decade, so if that’s going to happen soon, it would likely be happening already. (Is it?)

                                                                            1. 1

                                                                              That’s a good point. I thought it was to a small extent at least, but I can’t find details on such if they exist, so I might be wrong.

                                                                            2. 2

                                                                              AFAIK They are already doing that. Apple can probably do whatever they want on their own platform. There is no ARM police.

                                                                              1. 5

                                                                                There is no ARM police.

                                                                                There is. It’s called “Arm Ltd.” In order to add custom instructions (or design your own chip), you need an “architectural license.” Otherwise you must use the CPU core IP as-is (though you can of course add custom peripherals). Apple is one of the few companies with an architectural license.

                                                                                1. 1

                                                                                  Apple is one of the few companies with an architectural license.

                                                                                  Which isn’t too surprising since ARM was originally founded back in 1990 as a joint venture between Apple, Acorn and VLSI.

                                                                                2. 3

                                                                                  I am not sure if their license will allow them to use aarch64 name for such “extended architecture”. Also I do not think that they are interested in such extensions to the arch, as I think that they could easily push them into “standard” and then benefit from all the existing features of ARM community. They do not need Embrace, Extend, Extinguish as they are one of the big shareholders of ARM Holdings.

                                                                                  1. 2

                                                                                    They don’t use the aarch64 name though.

                                                                                    1. 1

                                                                                      Google for “A13 AMX” - which is their CPU instruction set extension for matrix operations.

                                                                                      1. 1

                                                                                        I find mostly French tanks so I couldn’t check if this is coprocessor or extension to the main CPU, but I believe that you may be right.

                                                                                        1. 1

                                                                                          It is not documented very well - mostly reverse engineered …

                                                                                          1. 1

                                                                                            Seems not bad and for sure not reverse engineered.

                                                                              1. 25

                                                                                I bought one last week and have used it for 7 days now. I was in an initial hype phase as well, but I am more critical now and doubting whether I should return it.

                                                                                Performance of native apps is as great as everyone claims. But I think it is a bit overhyped, recent AMD APUs come close in multi-core performance. Of course, that the Air works with passive cooling is a nice bonus.

                                                                                Rosetta works great with native x86_64 applications, but performance is abysmal with JIT-ing runtimes like the JVM. E.g. JetBrains currently do not have a native version of their IDEs (JVM, but I think they also use some other non-Java code) and their IDEs are barely usable due to slowness. If you rely on JetBrains IDEs, wait until they have an Apple Silicon version.

                                                                                Also, performance of anything that relies on SIMD instructions (AVX, AVX2) is terrible under Rosetta. So, if you are doing data science or machine learning with heavier loads, you may want to wait. Some libraries can be compiled natively of course, but the problem is that there is no functioning Fortran compiler supported on Apple Silicon (outside an experimental gcc branch) and many packages in that ecosystem rely on having a Fortran compiler.

                                                                                Another issue with Rosetta vs. native in development is that it is very easy to get environments where native and x86_64 binaries/libraries are mixed (e.g. when doing x86_64 development and CMake building ARM64 objects unless you set CMAKE_OSX_ARCHITECTURES=x86_64), and things do not build.

                                                                                Then Big Sur on Apple Silicon is also somewhat beta. Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well). When working longer, this issue disappears, but it’s annoying nonetheless.

                                                                                If you haven’t ordered one, it’s best to wait a while until all issues are ironed out. There is currently a lot of (justified hype) around Apple Silicon, but that doesn’t mean that the ecosystem is ready yet. Unless all you do is web browsing, e-mailing, and an occasional app from the App Store.

                                                                                Aside from this, I think there are some ethical (sorry for the lack of a better term) issues with newer Apple models. For example, Apple excluding their own services from third-party firewalls/VPNs, no extensibility (reducing the lifespan of hardware), and their slow march to a more and more closed system.

                                                                                Edit: returned and ordered a ThinkPad.

                                                                                1. 9

                                                                                  it’s best to wait a while

                                                                                  If you need a macbook now , for whatever reason, buying one with an Arm chip does sound the most future-proof option. The Intel ones will be the “old” ones soon, and will then be 2nd rate. It’s what happened with the PowerPC transition as well.

                                                                                  1. 2

                                                                                    If only there would be the Macs with 32GB RAM I would buy one as I was in need. However due to that, I bought 32GB 13” MacBook Pro instead. I will wait for polishing out the ARMs before next upgrade.

                                                                                    1. 1

                                                                                      From what I read, you get way more bang for your RAM in Apple processors. It’s all integrated on the same chip so they can do a lot of black magic fuckery there.

                                                                                      1. 1

                                                                                        In native applications - I am pretty sure that this works well, however as an Erlang/Elixir developer I use 3rd party GCed languages and DBs that can use more RAM anyway. However the fact that it is possible to run native apps from iOS and iPad could save some RAM on Slack and Spotify for sure.

                                                                                        1. 2

                                                                                          What I mean is, they probably swap to NAND or something, which could very likely be similar performance-wise to RAM you’d find on a x64 laptop (since they have a proprietary connection there instead of NVMe/M.2/SATA). Plus I imagine the “RAM” on the SoC is as fast as a x64 CPU cache. So essentially you’d have “infinite” RAM, with 16gb of it being stupid fast.

                                                                                          This is just me speculating btw, I might be totally wrong.

                                                                                          Edit: https://daringfireball.net/2020/11/the_m1_macs CTRL+F “swap”

                                                                                          1. 1

                                                                                            Just wondering if you had any take on this, idk if I’m off base here

                                                                                    2. 4

                                                                                      Lots of valuable insights here and I’m interested in discussing.

                                                                                      Performance of native apps is as great as everyone claims. But I think it is a bit overhyped, recent AMD APUs come close in multi-core performance. Of course, that the Air works with passive cooling is a nice bonus.

                                                                                      Sure, but the thing is that the AMD 4800U, their high-end laptop chip, runs at 45W pretty much sustained, whereas the M1 caps out at 15W. This is a very significant battery life and heat/sustained non-throttled performance difference. Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                                                      Rosetta works great with native x86_64 applications, but performance is abysmal with JIT-ing runtimes like the JVM. E.g. JetBrains currently do not have a native version of their IDEs (JVM, but I think they also use some other non-Java code) and their IDEs are barely usable due to slowness. If you rely on JetBrains IDEs, wait until they have an Apple Silicon version.

                                                                                      Yeah, I didn’t test anything Java. You might be right. You also mention Fortran though and I’m not sure how that matters in 2020?

                                                                                      Another issue with Rosetta vs. native in development is that it is very easy to get environments where native and x86_64 binaries/libraries are mixed (e.g. when doing x86_64 development and CMake building ARM64 objects unless you set CMAKE_OSX_ARCHITECTURES=x86_64), and things do not build.

                                                                                      This isn’t as big of a problem as it might seem based on my experience. You pass the right build flags and you’re done. It’ll vanish in time as the ecosystem adapts.

                                                                                      Then Big Sur on Apple Silicon is also somewhat beta. Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well). When working longer, this issue disappears, but it’s annoying nonetheless.

                                                                                      Big Sur has been more stable for me on Apple Silicon than on Intel. 🤷

                                                                                      If you haven’t ordered one, it’s best to wait a while until all issues are ironed out. There is currently a lot of (justified hype) around Apple Silicon, but that doesn’t mean that the ecosystem is ready yet. Unless all you do is web browsing, e-mailing, and an occasional app from the App Store.

                                                                                      I strongly disagree with this. I mean, the M1 MacBook Air is beating the 16” MacBook Pro in Final Cut Pro rendering times. Xcode compilation times are twice as fast across the board. This is not at all a machine just for browsing and emailing. I think that’s flat-out wrong. It’s got performance for developers and creatives that beats machines twice as expensive and billed as made for those types of professionals.

                                                                                      Aside from this, I think there are some ethical (sorry for the lack of a better term) issues with newer Apple models. For example, Apple excluding their own services from third-party firewalls/VPNs, no extensibility (reducing the lifespan of hardware), and their slow march to a more and more closed system.

                                                                                      Totally with you on this. Don’t forget also Apple’s apparent lobbying against a bill to punish forced labor in China.

                                                                                      1. 19

                                                                                        You also mention Fortran though and I’m not sure how that matters in 2020?

                                                                                        There’s really rather a lot of software written in Fortran. If you’re doing certain kinds of mathematics or engineering work, it’s likely some of the best (or, even, only) code readily available for certain work. I’m not sure it will be going away over the lifetime of one of these ARM-based notebooks.

                                                                                        1. 4

                                                                                          I’m not sure it will be going away over the lifetime of one of these ARM-based notebooks.

                                                                                          There will be gfortran for Apple Silicon. I compiled the gcc11 branch with support and it works, but possibly still has serious bugs. I read somewhere that the problem is that gcc 11 will be released in December, so Apple Silicon support will miss that deadline and will have to wait until the next major release.

                                                                                          1. 2

                                                                                            Isn’t Numpy even written in FORTRAN? That means almost all science or computational anything done with Python relies on it.

                                                                                            1. 6

                                                                                              No, Numpy is written in C with Python wrappers. It can call out to a Fortran BLAS/LAPACK implementation but that doesn’t necessarily need to be Fortran, although the popular ones are. SciPy does have a decent amount of Fortran code.

                                                                                            2. 1

                                                                                              Wow, who knew.

                                                                                              1. 23

                                                                                                Almost anyone who does any sort of scientific or engineering [in the structural/aero/whatever sense] computing! Almost all the ‘modern’ scientific computing environments (e.g. in python) are just wrappers around long-extant c and fortran libraries. We are among the ones that get a bit upset when people treat ‘tech’ as synonymous with internet services and ignore (or are ignorant of) the other 90% of the iceberg. But that’s not meant as a personal attack, by this point it’s a bit like sailors complaining about the sea.

                                                                                                Julia is exciting as it offers the potential to change things in this regard, but there is an absolute Himalaya’s worth of existing scientific computing code that is still building the modern physical world that it would have to replace.

                                                                                            3. 5

                                                                                              This is a very significant battery life and heat/sustained non-throttled performance difference.

                                                                                              I agree.

                                                                                              Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                                                              I am not sure what you mean. Modern Intel/AMD CPUs have AES instructions. AMD GPUs (including those in APUs) have acceleration for H.264/H.265 encoding/decoding. AFAIR also VP9. Neural depends a bit on what is expected, but you can do acceleration of neural network training, if AMD actually bothered to support Navi GPUs and made ROCm less buggy.

                                                                                              That said, for machine learning, you’ll want to get an discrete NVIDIA GPU with Tensor cores anyway. It blows anything else that is purchasable out of the water.

                                                                                              You also mention Fortran though and I’m not sure how that matters in 2020?

                                                                                              A lot of the data science and machine learning infrastructure relies on Fortran directly or indirectly, such as e.g. numpy.

                                                                                              I strongly disagree with this. I mean, the M1 MacBook Air is beating the 16” MacBook Pro in Final Cut Pro rendering times. Xcode compilation times are twice as fast across the board. This is not at all a machine just for browsing and emailing. I think that’s flat-out wrong.

                                                                                              Sorry, I didn’t mean that it is not fit for development. I meant that if you are doing development (unless it’s constrained to Xcode and Apple Frameworks), it is better to wait until the dust settles in the ecosystem. I think for most developers that would be when a substantial portion of Homebrew formulae can be built and they have pre-compiled bottles for them.

                                                                                              1. 1

                                                                                                Sorry, I didn’t mean that it is not fit for development. I meant that if you are doing development (unless it’s constrained to Xcode and Apple Frameworks), it is better to wait until the dust settles in the ecosystem. I think for most developers that would be when a substantial portion of Homebrew formulae can be built and they have pre-compiled bottles for them.

                                                                                                My instinct here goes in the opposite direction. If we know Apple Silicon has tons of untapped potential, we should be getting more developers jumping on that wagon especially when the Homebrew etc. toolchain aren’t ready yet, so that there’s acceleration towards readying all the toolchains quickly! That’s the only way we’ll get anywhere.

                                                                                                1. 16

                                                                                                  Well, I need my machine for work. So, these issues just distract. If I am going to spend a significant chunk of time. I’d rather spend it on an open ecosystem rather than doing free work for Apple ;).

                                                                                              2. 5

                                                                                                Sure, but the thing is that the AMD 4800U, their high-end laptop chip, runs at 45W pretty much sustained, whereas the M1 caps out at 15W. This is a very significant battery life and heat/sustained non-throttled performance difference. Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                                                                Like all modern laptop chips, you can set the thermal envelope for your AMD 4800U in the firmware of your design. The 4800U is designed to target 15W by default - 45W is the max boost, foot to the floor & damn the horses power draw. Also, the 4800U has a GPU…an 8 core Vega design IIRC.

                                                                                                Apple is doing exactly the same with their chips - the accounts I’ve read suggest that the power cost required to extract more performance out of them is steep & since the performance is completely acceptable at 15W Apple limits the clocks to match that power draw.

                                                                                                The M1 is faster than the 4800U at 15W of course, but the 4800U is a Zen2 based CPU - I’d imagine that the Zen3 based laptop APUs from AMD will be out very soon & I would expect those to be performance competitive with Apple’s silicon. (I’d expect to see those officially launched at CES in January in fact, but we’ll have to wait and see when you can actually buy a device off the shelf.)

                                                                                              3. 1

                                                                                                Edit: returned and ordered a ThinkPad.

                                                                                                That made me chuckle. Good choice!

                                                                                                1. 1

                                                                                                  You say that you returned and ordered a ThinkPad, how has that decision turned out? Which ThinkPad did you purchase? How is the experience comparatively?

                                                                                                  1. 2

                                                                                                    I bought a Thinkpad T14 AMD. So far, the experience is pretty good.

                                                                                                    Pros:

                                                                                                    • I really like the keyboard much more than that of the MacBook (butterfly or post-butterfly scissors).
                                                                                                    • It’s nice to have a many more ports than 2 or 4 USB-C + stereo jack. I can go places without carrying a bunch of adapters.
                                                                                                    • I like the trackpoint, it’s nice for keeping your fingers on the home row and doing some quick pointing between typing.
                                                                                                    • Even though it’s not aluminum, I do like the build.
                                                                                                    • On Windows, battery time is great, somewhere 10-12 hours in light use. I didn’t test/optimize Linux extensively, but it seems to be ~8 hours in light use.
                                                                                                    • Performance is good. Single core performance is of course worse than the M1, but having 8 high performance cores plus hyperthreading compensates a lot, especially for development.
                                                                                                    • Even though it has fans, they are not very loud, even when running at full speed.
                                                                                                    • The GPU is powerful enough for lightweight gaming. E.g., I played some New Super Lucky’s tale with our daughter and it works without a hitch.

                                                                                                    Cons:

                                                                                                    • The speakers are definitely worse than any modern MacBook.
                                                                                                    • Suspend/resume continues to have issues on Linux:
                                                                                                      • Sometimes, the screen does not wake up. Especially after plugging or unplugging a DisplayPort alt-mode USB-C cable. Usually moving the TrackPoint fixes this.
                                                                                                      • Every few resumes, the TrackPad and the left button of the TrackPoints do not work anymore. It seems that (didn’t investigate further) libinput believes that a button is constantly held, because it is not possible to click windows anymore to activate them. So far, I have only been able to reset this state by switching off the machine (sometimes rebooting does not bring bak the TrackPoing).
                                                                                                      • So far no problems at all with suspend/resume on Windows.
                                                                                                    • The 1080p screen works best with 125 or 150% scaling (100% is fairly small). Enabling fractional scaling in GNOME 3 works. However, many X11/XWayland applications react badly to fractional scaling, becoming very blurry. Even on a 200% scaled external screen. Also in this department there are no problems with Windows, fractional scaling works fine there.
                                                                                                    • The finger print scanner works in Linux, but it results in many more false negatives than Windows.

                                                                                                    tl;dr: a great experience on Windows, acceptable on Linux if you are willing to reboot every few resumes and can put up with the issues around fractional scaling.

                                                                                                    I have decided to run Windows 10 on it for now and use WSL with Nix + home-manager. (I always have my Ryzen NixOS workstation for heavy lifting.)

                                                                                                    Background: I have used Linux since 1994, macOS from 2007 until 2020, and only Windows 3.1 and briefly NT 4.0 and Windows 2000.

                                                                                                  2. 1

                                                                                                    Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well).

                                                                                                    Sleep seems to be broken on the latest MacOS versions: every third time I close the lid of my 2019 mac, I’m opening it later only to see that it has restarted because of an error.

                                                                                                    1. 1

                                                                                                      Maybe wipe your disk and try a clean reinstall?

                                                                                                  1. 17

                                                                                                    From my perspective, these kinds of behaviors seem to creep up more on culture, person and practices threads. Could we try marking those tags as inactive and see if the site behavior improves/number of flags on comments drops?

                                                                                                    1. 12

                                                                                                      I expect for at least a while they’d be submitted and tagged with just programming. How much content relevant to culture/person/practices would prompt you to remove such a story? Would your answer change if all the comments are about that aspect of the story? Or if it’s the elephant in the room?

                                                                                                      Some test cases, in the hopes that they’re useful for figuring out how to draw a line:

                                                                                                      • A famous programmer dies
                                                                                                      • A famous programmer joins or departs a project and writes about technical issues they will/did encounter
                                                                                                        • and they were forced out for discriminatory behavior, though the post never mentions or refers to it
                                                                                                      • A study (academic or ad-hoc) of bug rate by language
                                                                                                        • and by development methodology (agile, scrum, tdd, etc.)
                                                                                                        • Would your answer to the previous change if it was a one-sentence mention of a correlation vs half the study?
                                                                                                        • How about if it was a one-sentence mention suggesting it for future work?
                                                                                                      • A retrospective on the author’s contributions to an open source project
                                                                                                        • that was paid work for their employer
                                                                                                        • and the employer is accused or held liable for discriminatory behavior, is a political party, or is a political advocacy group
                                                                                                        • Or the author thanks a business like Patreon or Github Sponsors for making their work possible
                                                                                                      • A blog post about the author’s first contribution to fix a bug in an open source project
                                                                                                        • and the PR is rejected
                                                                                                        • and the submitter believes it’s for discriminatory reasons
                                                                                                        • or the reasons given are explicitly discriminatory
                                                                                                      • A blog post about x86 minutiae from a programmer who famously advocates for discrimination
                                                                                                        • A blog post about why nobody should contribute to that famous author’s project because of their political views
                                                                                                      • A blog post about using ML to locate humans in video
                                                                                                        • in surveillance video from a military drone
                                                                                                        • that refers to the military or political conflict it will be or was used in

                                                                                                      And of course this all comes up again in comments.

                                                                                                      How do you feel about culpability? I’m thinking of sayings like “the standard you walk past is the standard you accept” that cast a failure to act as a position in favor of the status quo.

                                                                                                      1. 4

                                                                                                        I personally feel that either all of your “discriminatory behavior” test cases should be in-scope for discussion within this community, or there should otherwise be an outright ban on anything that breaks the meta barrier (or not-strictly-about-hard-tech barrier), emulating something akin to a dry technical journal with a strictly-moderated comments section.

                                                                                                        But let’s be honest: there’s already precedent on this website for discussion of topics that intersect with computing; I’ve seen a number of well-upvoted/discussed stories including:

                                                                                                        • blog posts and updates from prominent community members about things happening in their lives
                                                                                                        • posts discussing history and context of open source projects
                                                                                                        • news stories about prominent community members
                                                                                                        • (this one may sound a little salty, but:) fluff pieces, rants, and straight-up advertisements written by darlings of the community
                                                                                                        • meta-tagged posts such as this one that explicitly encourage us to think critically about the community

                                                                                                        So IMO it’s at best not self-reflective and at worst intellectually dishonest to value critical engagement with topics that intersect in interesting ways with computing, except for the cases in which those intersectional topics touch on problems of discrimination and representation in tech. A strong message is sent to newcomers and passersby when they hop into the comments section, and see that many of the well-upvoted comments on this website are in fact hostile and intended to trivialize and demean when the topic at hand is actually critical to community health. Whether it’s meant to or not, this sends an unkind message to marginalized members of this community, and it will – as it has – homogenize participation on this platform over time.

                                                                                                        1. 5

                                                                                                          Good test cases. Ones I’ll point at in particular:

                                                                                                          and they were forced out for discriminatory behavior, though the post never mentions or refers to it

                                                                                                          If the post is about the technical issues they encountered, I think it should stay. Them getting kicked out (or brought in!) due to discriminatory behavior is drama and only leads to poor discussion.

                                                                                                          and the employer is accused or held liable for discriminatory behavior, is a political party, or is a political advocacy group

                                                                                                          We had this happen with a newbie lobster who worked at Palantir–they were roasted with pointy questions before they could really represent their work. That was super shitty.

                                                                                                          A blog post about the author’s first contribution to fix a bug in an open source project

                                                                                                          I think this is fluff, and attracts fluff, unless the framing is explicitly “here is this super interesting technical thing that also happens to be my first contribution back”. Further subpoints all boil back down to drama.

                                                                                                          A blog post about x86 minutiae from a programmer who famously advocates for discrimination

                                                                                                          If it’s valid x86 minutiae it should be tagged assembly, and commentary about their political hobby horses is just as relevant as speculation on what configuration of genitals they prefer during intercourse. Users repeatedly dragging that up she be flagged and if needed encouraged to find communities with a more humanitarian focus elsewhere.

                                                                                                          ~

                                                                                                          I’m thinking of sayings like “the standard you walk past is the standard you accept” that cast a failure to act as a position in favor of the status quo.

                                                                                                          I think those sayings are troublesome because they by definition assume bad faith on the part of the people walking by. Also, I dislike them because they remove all shades and alternative interpretations of interaction in favor of plain “you’re either with us or against us”…and if folks keep asking for that sort of conflict, I think they’re gonna be rudely surprised by the outcomes they get. Blind tribalism doesn’t lend itself to healthy discussion.

                                                                                                          1. 5

                                                                                                            If the post is about the technical issues they encountered, I think it should stay. Them getting kicked out (or brought in!) due to discriminatory behavior is drama and only leads to poor discussion.

                                                                                                            I infer from this that you think drama should be removed. Can you define that term? Is it a heckler’s veto, such that any reading of “drama” into a post means it should be removed? Same for “fluff”? Especially with those being new, unspecific terms, I think the questions about where “how much content” mean a comment or story should be removed are vital.

                                                                                                            1. 5

                                                                                                              “Fluff” is the easier one to define here: pieces that are based in exceptionally common experiences but which are also going to probably get sympathetic upvotes. The example of “my first PR” is fluff because a) a lot of people have their first PR and b) what kind of asshole would flag such a thing. Fluff tends to have an advantage against other content in any memetic ecosystem lacking explicit pressures against such simple content. That simplicity is also why I have the exception for framing it as a real technical issue that just so happens to be somebody’s first PR.

                                                                                                              “Drama” is much harder, but as I use it: content that is significantly about the internal politics or disagreements inside some community. A test might be “if you replaced the people with other people could the problem be made to go away?” Donglegate could have been resolved with either party being replaced, Heartbleed could not. I believe that the corollary to this is that any discussion involving drama will ultimately involve calls to remove or reprogram other humans, and thus are inherently corrosive for a community such as ours to engage in.

                                                                                                          2. 1

                                                                                                            I distilled these examples to express a couple problems with defining and enforcing topicality.

                                                                                                          3. 5

                                                                                                            +1 from me.

                                                                                                            I’ve had a number of long discussions with 35 recently over whether retiring those tags would skew the amount of incidents back down to normal.

                                                                                                            1. 3

                                                                                                              I agree with you those tags lead to toxic discussion more than others, but I think it’s worth emphasizing that a lot of @itamarst’s examples are in a person thread.

                                                                                                            1. 15

                                                                                                              While suckless as a way to build software is definitely interesting (and a lot of the benefits of suckless come from disregarding edge cases, internationalization and accessibility), I’d argue that suckless as a project is something one should handle really carefully.

                                                                                                              They’re doing literal torch hikes through southern Germany at their conferences, fighting online against “cultural marxism”, and their mail server has the hostname “Wolfsschanze” (see: https://twitter.com/pid_eins/status/1113738766471057408)

                                                                                                              I recommend reading this thread (with a suckless.org developer with enabled hat speaking officially) and looking at this photo from one of their conferences.

                                                                                                              1. 20

                                                                                                                The topic pops up here and there, and one should always consider that Lennart Poettering used this bait to easily escape from a discussion about his software that I personally think should take place. Suckless is not directly a coherent group and more of a group of like-minded individuals, so I’m careful to speak “for” the project even though I’m first chair of the legal entity suckless.org e.V..

                                                                                                                What I can say is that we are probably one of the very very few software projects left that do not politicize. We have members from all political spectrums, but make it work, because we only discuss software and nothing else. Those disagreeing with us or unaccustomed to non-political software projects try to put us into a corner, but it makes no sense when it is used to disregard the suckless philosophy itself, which is non-political.

                                                                                                                Torch hikes are nothing unusual in Germany and there was no political intent behind it. Though I do understand now that it might send a different message worldwide, I expect more cultural sensibility from every observer before “calling us out” for allegedly re-enacting nazism or celebrating a Charlottesville march, which is a ridiculous assessment.

                                                                                                                1. 23

                                                                                                                  One should always consider that Lennart Poettering used this bait to easily escape from a discussion about his software that I personally think should take place.

                                                                                                                  Perhaps, but I don’t think calling out getting emails from a wolfsschanze host is that unreasonable to be honest; as I mentioned in my other post I’m not going to attach far-fetched conclusions to it but I do find it in pretty bad taste. At any rate, to ask it plainly, what’s the deal that?

                                                                                                                  1. 3

                                                                                                                    There is no such thing as “non-political”, because we live in a society with power imbalances. Therefore not taking an explicit political stance, translates to implicitly supporting the status quo, whatever that status quo is in a given society at a given time. You’ll find that people in underrepresented demographics will largely avoid your project as a result, regardless of the political views among members of your project.

                                                                                                                    If supporting the status quo is what you intend to do, then that is one thing. But please stop presenting it as “non-political”, because that is simply not the reality of it. It only looks non-political if you yourself are in a position where the status quo benefits you. Which I am also - so this is not a personal accusation, to be clear. But it is something you need to be aware of.

                                                                                                                    1. 19

                                                                                                                      not taking an explicit political stance, translates to implicitly supporting the status quo

                                                                                                                      No no no, I cannot agree with that. Let’s take an example. I’m working on a crypto library, that on many aspects is very close to the Suckless ideals: it’s in C, it’s small, it’s easy to integrate into other projects… One of the motivations for it was to fight unnecessary complexity. A fairly political goal if you ask me: if software becomes as simple as I think it can (and should) be, the changes could be felt throughout the global economy itself.

                                                                                                                      My project also has an implicit endorsement of the status quo: it is written in English, and I have no intention to translate the documentation, or even the web site to other languages. Not even French, my native language. Sorry, you need to learn English to use my project. That’s kind of an implicit endorsement of US hegemony. Not that I’m very happy about that, but not fighting that fight does make me reinforce the ubiquity of the English language.

                                                                                                                      But there’s no way my project can have a stance on everything. Its stance on many many subjects is really neutral. It does not fight nor reinforce the status quo. Veganism? Patriarchy? White supremacy? I hardly have a community to speak of, there’s just not enough people to warrant something like a code of conduct. That does not mean my project implicitly rejects vegan transgender black women. In fact, I do hope they’ll feel as welcome as anyone else. And right now, I believe being nice to whoever contacts me is enough.

                                                                                                                      1. 8

                                                                                                                        I couldn’t have put it better, thanks for sharing your thoughts. I always like to consider the example of Chemistry: In the 19th and 20th century, German scientists were leading in chemistry and most papers were published in German. Chemistry students were more or less forced to learn German to understand these papers, and German became the lingua franca of Chemistry, which has changed to English though.

                                                                                                                        In computer science, English is the lingua franca. I don’t think it’s exclusionary to only offer software documentation and code comments in English.

                                                                                                                      2. 8

                                                                                                                        That is a good point and I understand what you mean with that. For our conferences, we actually offer those who are unable to attend due to financial reasons to pay their travel expenses and accomodation for them, which was greatly appreciated especially by younger programmers who often don’t have the means to fund such a travel.

                                                                                                                        Apart from income differences, that might be a deciding factor being unable to attend a conference and meeting like-minded people, I see no other factors that might hinder someone from joining us. You basically only need an internet connection and a computer. The computer doesn’t even need to be that fast, unlike if you, for instance, intended to work with deep learning software.

                                                                                                                        And if you still criticize the conferences for being exclusionary in some way: Most communication takes place on a mailing list and IRC, many people use pseudonyms. Factors like race, country of residence, gender are thus irrelevant and even non-determinable, if you choose to, and the development on mailing lists and IRC is the main way development happens and there’s no need to do anything else to partake or make submissions.

                                                                                                                        So, again, I know what you mean, but suckless is not an example for a project supported by the status quo. Most people disregard suckless as too extreme in terms of software philosophy and conclude that we would also be extreme in other areas of life, but suckless, for me, is software zen, and everyone is welcome to adopt this philosophy.

                                                                                                                        1. 5

                                                                                                                          Factors like race, country of residence, gender are thus irrelevant and even non-determinable, if you choose to, and the development on mailing lists and IRC is the main way development happens and there’s no need to do anything else to partake or make submissions.

                                                                                                                          This is a common point of view among those in privileged demographics. However, it is also a misunderstanding of how underrepresented people in demographics actually choose where to hang around and contribute, and why.

                                                                                                                          Imagine for a moment that you are someone in a demographic who’s frequently a target of harassment. The exact demographic doesn’t matter much - maybe you’re black, or you’re a woman, or you’re transsexual, or whatever else. But for the sake of the example, imagine that you are a woman.

                                                                                                                          Now, there are two different communities for you to choose from:

                                                                                                                          1. A community that says “we don’t police member’s politics, this is purely a tech project”.
                                                                                                                          2. A community that says “we actively welcome women”.

                                                                                                                          Where are you going to feel safer? In the second community - because there, it’s clear that if someone finds out you’re a woman, them harassing you over it isn’t going to be tolerated and the harasser is going to be thrown out.

                                                                                                                          In the first community, you just kind of have to stay quiet about your identity, have everyone assume that you’re a guy, and hope that no-one finds out the truth. If they do - maybe there’s some persistent stalker following you around and posting about you in every community you join - you can basically predict ahead of time that harassment and other discriminatory behaviour is not going to be acted upon, because “people’s own politics are not policed”.

                                                                                                                          In a way, there are parallels here to how gay people are “tolerated” in many countries. It’s “fine so long as you don’t bother me with it”, which effectively means that you cannot speak about it publicly or have a public relationship with someone of the same sex, because then the cover falls away and you are no longer “okay”, because your identity can no longer be ignored. Harassment (and often violence) promptly follows.

                                                                                                                          “Don’t ask, don’t tell” policies like this don’t make for healthy, diverse environments. They make for environments in which the status quo is preserved, and where the only way to be vaguely safe as a minority is to never tell anyone that you don’t fit into that status quo. This is not inclusive, and it absolutely does support the status quo. Those who fall outside of it will silently move on to healthier communities.

                                                                                                                          I would like it if “who you are doesn’t matter, it’s about the project” were the reality, I really would. But that just isn’t how things work by default in a society with power imbalances, and the only way to get there is by actively enforcing it - and that means taking a political stance, one that disavows discriminatory behaviour and harassment.

                                                                                                                          1. 12

                                                                                                                            Now, there are two different communities for you to choose from:

                                                                                                                            1. A community that says “we don’t police member’s politics, this is purely a tech project”.
                                                                                                                            2. A community that says “we actively welcome women”. Where are you going to feel safer?

                                                                                                                            I don’t know how the suckless community is, but I am convinced that, if I had a dime for every company, group or project that claimed to “actively welcome women” or “promote equal opportunity for everyone” or “have a zero tolerance” towards discrimination, sexual harassment or any other one of the multitude of abhorrent behaviours that plague our industry – and then turned out to be cesspools of prejudice and discrimination, I would be so outrageously rich that even thinking about it is embarrassing.

                                                                                                                            (FWIW, in addition to witnessing it enough times that it’s part of why I seriously contemplated switching careers at one point, I have some first-hand experience with some of that: my most useful skill, career-wise, has been an impeccable accent. Slightly Irish-sounding (which lots of folks in the US seem to fetishize for some reason), which I developed purely by accident (I’m from nowhere near Ireland, I’ve never been there, and I am not a native English speaker) and is extremely embarrassing every time I’m talking to someone who has a real Irish accent. I certainly had it easier than my black or hispanic colleagues – most Western managers of various importance in the corporate hierarchy could immediately identify them as worthy of contempt, whereas in my case it could take weeks before they realized I’m not a white expat, just some Eastern European programmer.

                                                                                                                            Edit: in case anyone’s wondering – the reason why I can be so light-hearted about it is that, for better or for worse, this experience has been largely confined to the workplace, after-work drinks, conferences and the like. I got to live with it for like 40 hours a week at most, and never really got a taste of it before well into adulthood. I always had alternatives and always had refuge – I could always put up with it on my own terms, which most people can’t)

                                                                                                                            Coming from a culture whose closet is not devoid of skeletons in this department, either, I certainly agree that the mere act of not discussing race, or gender, or ethnicity is in itself a privilege that not everyone has. And that it’s up to every one of us to actively fight discrimination, and to make the world safer and more inclusive for those whose voices are silenced by intolerance. But I don’t think it’s reasonable to ask people to integrate that in every single thing they do. Even activists don’t fight the good fight 24/7, I don’t think it’s unreasonable that some people choose to do it only to a limited extent, or in a possibly misguided way, as part of their hobby project.

                                                                                                                            1. 9

                                                                                                                              I might’ve been a bit unclear. A don’t-ask-don’t-tell approach can be taken by members, if they prefer (many communities don’t provide that luxury and e.g. require clear-name-contributions), but doesn’t have to be. We just don’t care about genders or other aspects other than your coding skills. I see that you have a different opinion on this, which is cool, but the suckless philosophy does not extend beyond software aspects and I personally (not speaking for the group) don’t see a reason to extend that.

                                                                                                                              1. 5
                                                                                                                                1. A community that says “we don’t police member’s politics, this is purely a tech project”.
                                                                                                                                2. A community that says “we actively welcome women”.

                                                                                                                                The two may not be mutually exclusive. Although there’s certainly a huge overlap, there’s a difference between advocating the revocation of women’s right to vote, and actually harassing women in a particular group, be it an open source project or a chess club.

                                                                                                                                A president of a chess club, or a maintainer of an open source project, can hardly be expected to be aware of the political views of the members, no matter how extreme. He could pry, but that would be uncomfortable for many people, and ultimately exclusionary. We could do it anyway, and define the range of acceptable political opinions, and exclude the outliers. We could exclude traditionalists, or we could exclude gay marriage supporters. We could exclude white supremacists, or we could exclude black panthers sympathisers.

                                                                                                                                In my opinion this would be neither ideal nor possible. As long as people stay courteous and focus on working towards whatever common goal the group has, we could actually have, say, gay and homophobic people working together. So we probably want to define a range of acceptable behaviours instead. For instance, revealing your sexual preferences is generally acceptable (unless maybe you’re too loud about this), and revealing your contempt for people who don’t share that preference is generally not.

                                                                                                                                That’s what codes of conduct ultimately do: they don’t talk about the politics one might have outside of the group, they define a range of acceptable behaviours within the group. Yes, that range will tend to filter out people with some particular political opinions. Few white supremacists will follow a black maintainer. But I would think real hard before I make that filter explicit.

                                                                                                                                I’ve seen it done, and it’s not pretty. I’ve heard of someone being disinvited from some conference because of their political beliefs, even though they (allegedly) never let them seep through or ever behaved inappropriately. I have also heard of someone being fired over their sexual practices (at the behest of SJW, ironically). And at the same time, some people who seem to engage in genuinely harmful behaviour (such as straight up sexual harassment) are not excluded. My suspicion? Enforcement goes after the easy targets, instead of going after the important ones.

                                                                                                                                1. -5

                                                                                                                                  we could actually have, say, gay and homophobic people working together.

                                                                                                                                  Honestly, this free speech absolutism is whack and that’s why I’m out.

                                                                                                                                  You don’t know what the fuck you’re allowing. I do - you’re allowing someone who literally spreads hate to walk into work, meekly get some shit done, then go home to post on the internet how trans people are all pedophiles and should be killed.

                                                                                                                                  Fact is, you can’t divorce your life from politics because where many of us stand, all minorities, live under the continuous threat that we’ll be murdered, denied service, beaten and reviled all because some free speech absolutist like you envisions a future where racists and their victims can work side by side.

                                                                                                                                  My community just had their 12th death. Death because people like you continually give deference to allow our killers to bask in their hate speech until one of them spots us and brutally kills us.

                                                                                                                                  You enable this. I’m so happy (not) to be the sacrificial lamb for your perverse ideology.

                                                                                                                                  1. 2

                                                                                                                                    we could actually have, say, gay and homophobic people working together.

                                                                                                                                    Honestly, this free speech absolutism is whack and that’s why I’m out.

                                                                                                                                    Who said anything about free speech? I never said hate speech should be allowed. Actually, I do believe we free speech should have limits (though I’m not sure exactly what those should be), and people who cross those limits should be punished.

                                                                                                                                    The question is who should punish them, and how. Forums can (and most probably should) ban hate speech however they can. Police and Judges could intervene whenever appropriate. The worst offenders could be sent to jail.

                                                                                                                                    Wholesale ostracism though? Exclusion from all groups, not just wherever they spread their filth? That’s a death sentence: no job, no home, no shelter. Are you prepared to follow through all the way? (Not a rhetorical question: sometimes, killing your enemy is the right thing to do. But this question is so fraught with self serving cognitive biases that one must be very careful about it.)

                                                                                                                                    Then there are false positives. The guy who was fired over his sexuality? He practised BDSM. One way of putting it is that he liked to whip bound women. When he was outed, there was an SJW outcry about him being some twisted archetype of patriarchy that should be removed from any public position.

                                                                                                                                    I don’t know the guy, I haven’t investigated, so I cannot presume. I’m not even certain this story is even true. But I guess this may have been a huge misunderstanding. See, done properly, BDSM is very careful about safe words, physical and psychological safety… everyone is supposed to enjoy this, including (perhaps even primarily) the bound and gagged “victim”. Being a good dom typically requires empathy and respect for their sub. Pretty far from the simplistic image of the misogynistic man taking pleasure from the suffering of women.


                                                                                                                                    Going back to gays and homophobic working together, that probably requires that they are mutually unaware of their position. It’s when they do become aware of their position that we have a problem, and the group may have to make a choice. My first step would be something like “you don’t like them being gay? deal with it or get the fuck out”. If it’s just gay people being uncomfortable, we may need to know why. If it’s because the other dude displayed an homophobic attitude within the group, that’s pretty obvious grounds for exclusion. If it’s because gay people learned of his views outside the group, this is more delicate, and I honestly have no right answer.

                                                                                                                                    The problem is made even harder because actual bullying, embarrassment, and other inappropriate behaviour within a group, are often hard to see for anyone but the victim. Hence the temptation to rely on more visible, but less reliable, external signs.

                                                                                                                                    For instance, let’s imagine: religious people and atheists working together in the same group. One atheist have written in their blog about how religion is stupid, unfounded, and how religious people are either critically misinformed, or just plain delude themselves. Oh and by the way if there is a God, it’s morals are highly questionable at best. So there we go: no personal insult, but a harsh criticism and a good dose of blasphemy.

                                                                                                                                    Should we exclude this atheist from a chess club because some religious people in that club feel uncomfortable being next to someone who has written a blasphemous pamphlet? Should we exclude the religious people from the club because wearing a cross, a star, or a scarf makes the atheist uncomfortable? Depending on who you ask, you’ll have very different answers.

                                                                                                                            2. 5

                                                                                                                              On the other hand, I don’t think it’s realistic to expect every project to look in depth at difficult social problems and form some sort of consensus on how to best deal with it.

                                                                                                                              You’ll find that people in underrepresented demographics will largely avoid your project as a result

                                                                                                                              Why would that be the case?

                                                                                                                              1. -4

                                                                                                                                On the other hand, I don’t think it’s realistic to expect every project to look in depth at difficult social problems and form some sort of consensus on how to best deal with it.

                                                                                                                                I think that’s entirely reasonable. This is pretty much the basis of community management in general. It doesn’t even need to be done by the core developers, but someone in the community needs to do it, if you want a healthy community.

                                                                                                                                Why would that be the case?

                                                                                                                                Because they know that their safety is not assured in communities that refuse to take an active stance against bigotry of various kinds. I’ve gone into more detail about this in this other subthread.

                                                                                                                                1. 4

                                                                                                                                  Because they know that their safety is not assured in communities that refuse to take an active stance against bigotry of various kinds.

                                                                                                                                  But there is a difference between belief and action. If someone is actually doing something bad within the project then obviously that’s an issue. If someone just believes something you disagree with (whether you label it bigoted or not) then refusing to work with them in a non-political atmosphere just makes you seem like a bit of a dick, IMO.

                                                                                                                              2. -3

                                                                                                                                There’s no such thing as “non-political” software projects because any political actor can decide that the way your software project run things is bad and should be made to change. And if you resist this, you find yourself in a political conflict, even if you didn’t want to be.

                                                                                                                                1. 1

                                                                                                                                  Why would you care what a political actor thinks about your free software project? Do you mean an actual national politician? Why would they be concerned with a free software project?

                                                                                                                                  1. 2

                                                                                                                                    No, anyone trying to argue that a software project should change their practices for political reasons is a political actor with respect to software, not just national politicians. Tech industry activists are political actors. joepie91 in this thread is a political actor. I’m a political actor too, for trying to prevent other political actors from carrying out their will.

                                                                                                                              3. -1

                                                                                                                                What are you doing to keep this kind of toxic behaviour from forming inside of the suckless communities you participate in?

                                                                                                                                You have not denied that these people exist in your community. How are they not a problem for you?

                                                                                                                                1. 4

                                                                                                                                  Calling people toxic, I think, is the wrong approach. What matters is how people behave in the context of the community. I couldn’t care less about their private political/social/other endeavours as long as it doesn’t affect their actions within the community.

                                                                                                                                  I don’t know why there is such a push to politicize software projects, from the inside and outside. It may make something look more homogenous on the outside, but I believe it mostly creates social stress and shifts the focus on issues that shouldn’t be a problem in the first place. But this is just my opinion, and I don’t think there’s a true or false answer to that. It heavily depends on your Weltanschauung.

                                                                                                                                  1. 1

                                                                                                                                    I’m sorry, my first approach was a bit antagonistic and too political because I tried to keep my questions short.

                                                                                                                                    People sometimes express their political ideologies in behavioural ways, which might cause exclusion and secularity in the communities that they take part in. I haven’t been much in contact with the suckless community, although I have used and I respect the software and the philosophy, but I have seen communities suffer this. I have no prejudice, but toxic (extreme, hateful) ideologies do lead to toxic behaviour, especially in like-minded groups where it can be cultured. This is why people feel the need to keep them from spreading to their own group.

                                                                                                                                    Have you noticed any exclusive or secular behaviour in the suckless communities that you take part in? If yes, what have you been doing to counter it?

                                                                                                                                    1. 3

                                                                                                                                      Have you noticed any exclusive or secular behaviour in the suckless communities that you take part in? If yes, what have you been doing to counter it?

                                                                                                                                      No, I’ve never seen such secular behaviour like that. The conferences we organize have always been very harmonic and there was never such a push or even a culturation. Thanks though for elaborating what you meant, and I have to say that I’ve seen this problem occuring within other communities. I am and will be very careful that this won’t happen within our community.

                                                                                                                              4. 21

                                                                                                                                I was subscribed to the suckless mailing list for a long time (though no longer, simply out of disinterest), and never had the impression I was dealing with a group of extremists (other than a rather extreme take on software). I don’t recall any political discussion off-hand, and would certainly have unsubscribed if people started ranting about “cultural Marxism” and the like.

                                                                                                                                I read the Lobsters thread you linked and there are many things I personally don’t agree with, but I also find it’s a lot more nuanced than what you’re suggesting (specifically, there was a lot of confusion what was even intended with “Cultural Marxism”). I saw that on HN you (or someone else?) linked to an old tweet of yours that screenshotted just the initial “Cultural Marxism” mention of FRIGN, and I think that’s unfairly out of context. That’s not a defence of the contents if his posts, only a defence of treating people with fairly and with kindness.

                                                                                                                                I find putting the picture of the torches next to literal Nazis and the “Unite the Right” rally incredibly tasteless and offensive. Note the suckless event happened before the Charlottesville march (not that it really matters). [edit: incorrect, see follow-up]. I’ve done torch hikes – they’re actually used to celebrate the end of Nazi occupation in my home town every year and participated regularly. I’ve also done them with scouts just for the fun of it. Maybe some day someone will dig up a picture of that too and put it next to a bunch of Nazis to prove a point… I’m very disappointed anyone would try to make a point like that, here or elsewhere. This part of your post in particular is really bad in many ways IMHO; it’s really not acceptable to just sling around grave insinuations like that based on a friggin’ contextless photo of what is almost certainly just a harmless social event.

                                                                                                                                The mail server belongs to an individual (@FRIGN here). I agree it’s in very bad taste, offensive, and that Poettering was completely right in calling that out, but it’s hardly proof that “they’re a bunch of Nazis”. I find the jump from “edgy hostname” to “literal neo-Nazis” a bit of a leap.


                                                                                                                                I doubted for a long time if I should post this reply as it has the potential to spark a long heated discussion, but I find public casual comparisons to Nazis in particular serious enough to warrant something of a rebuttal.

                                                                                                                                1. 6

                                                                                                                                  Note the suckless event happened before the Charlottesville march (not that it really matters).

                                                                                                                                  I just want to comment on this one factual point, according to the suckless website this event happened in September 2017, just a couple of weeks after Charlottesville.

                                                                                                                                  https://suckless.org/conferences/2017/

                                                                                                                                  I do think the proximity in time to the Unite the Right rally is important, especially given the insistence that they were just enacting a German cultural practice.

                                                                                                                                  1. 6

                                                                                                                                    Oops, I checked the website and I misread that date as being on “2017-01-03”, instead of “2017-09-(01-03)”. How silly 😅🤦‍♂️

                                                                                                                                    I’m not sure it matters all that much though; it still seems incredibly tenuous at best. This happened on the other side of the world and I’m not sure if the entire world should tip-toe around sensitive topics in the United States. Were these people even aware of Charlottesville? And to what degree? Me, personally, I mostly stopped following US news since the 2016 election as I find it emotionally draining and serving little purpose as it’s not in my power to do something about anyway.

                                                                                                                                    Either way, I’d sure like to see some more evidence exactly because I take it serious: you just don’t go around insinuating people of such serious things with such slim “surely it can’t be coincidence…” type of stuff.

                                                                                                                                    1. 31

                                                                                                                                      I was at the torch hike and hadn’t even heard of the Charlottesville marches then. When I heard the accusation that we in some way celebrated it, which would make no sense in the context of a software conference, I first had to look up what they were.

                                                                                                                                      The thing is, Americans tend to overestimate the importance of domestic events like the Charlottesville marches and think that nothing happens in the whole world and, e.g., we Germans are just sitting at home and waiting for something to happen in the USA to witness it.

                                                                                                                                      The truth, and I think everyone would agree that this also makes much more sense, is that torch hikes are perfectly normal in Germany. I have an understanding for this cultural misunderstanding, and I’ve been guilty of those, as well, but it doesn’t help when one continues to spread this nonsense that this torch hike was some political event every time suckless is discussed here.

                                                                                                                                      To give an example for how normal torch hikes in Germany are, there is a so-called Sommertagszug in the Kurpfalz which also involves torch hikes at night. They are also offered by tourist organizations, e.g. Breitbach Klamm.

                                                                                                                                      1. 8

                                                                                                                                        What’s with the mail server host name though? Do you think that’s fine?

                                                                                                                                        1. 4

                                                                                                                                          It bothers me that he is actively ignoring this question and by saying nothing, he is saying enough.

                                                                                                                                        2. 2

                                                                                                                                          As an American, thanks for sharing your perspective. It makes me wonder if the Internet, and particularly social media, make it too easy to carelessly make connections between things that should remain disconnected. Maybe Facebook’s stated mission of making the world more connected (whether or not that’s their real mission) isn’t a totally good thing.

                                                                                                                                          1. 5

                                                                                                                                            It definitely comes at a cost. Still, as I could see from my own experience, after a few years one gets more careful with culture-relative judgements. There are still many things Americans do that I don’t quite understand or find interesting.

                                                                                                                                            To give an example, I found out a few years ago that the German “mhm” (i.e. the expression to acknowledge you are listening to someone while he speaks) is often interpreted by Americans as a “huh?”. You could imagine how much confusion that caused.

                                                                                                                                            Cultural differences are valuable, though, and I would not want to miss them, even if they become troublesome. I can imagine an American coming to Germany to experience a torch hike and liking it.

                                                                                                                                            1. 0

                                                                                                                                              To give an example, I found out a few years ago that the German “mhm” (i.e. the expression to acknowledge you are listening to someone while he speaks) is often interpreted by Americans as a “huh?”. You could imagine how much confusion that caused.

                                                                                                                                              I have never in my life seen or heard “mhm” interpreted as “huh?”, and while I’m just one American and this is anecdotal I’ve lived in three fairly distinct regions of the USA.

                                                                                                                                              1. -1

                                                                                                                                                German “mhm” is very distinctly different to American “mhm”. I wouldn’t know how to describe it in words, though.

                                                                                                                                                1. 0

                                                                                                                                                  It’s it very distinct from the British “mhm”?

                                                                                                                                          2. 1

                                                                                                                                            Going on a torchlit hike at night sounds fun to me in the abstract, and also like the sort of activity that could hardly be unique to any one place, time, or culture. For ages before the invention of electric flashlights, how else were human beings supposed to light their way when walking around at night, wherever in the world they happened to be? I was unaware that some people associated the practice of going on torchlit hikes with specifically the NSDAP (or maybe just going on a torchlit hike while being an ethnic German??) until I saw people mentioning it in the context of suckless. Even if it’s true that the historical Nazis practiced torchlit hikes (which I assume is true, because I think it would be very easy for any group in human history to do so), I don’t think that confers any obligation on people alive today to refrain from it, any more so than Adolf Hitler’s famous vegetarianism confers any obligation on people today not to be vegetarians.

                                                                                                                                            1. 3

                                                                                                                                              I agree. I’m pretty well read on the topic, including having read Shirer’s “Rise and Fall of the Third Reich,” and I hadn’t heard about the association between torchlit hikes and Nazis before it was brought up in the context of suckless either. If I’m actually educated on the topic and still didn’t know about it, how could I really expect others to know about the association?

                                                                                                                                              Personally, a torchlit hike sounds like a blast to me. If the opportunity presented itself to me, I would absolutely participate.

                                                                                                                                              I agree with others in this thread that people are generally way too quick to bring up Nazi associations. I like to think I’m not naive about it either, since there are trolls and Nazis online that like to play these kinds of games. But I personally expect some pretty firm evidence before I’m willing to entertain Nazi accusations seriously. It’s a pretty serious thing to say.

                                                                                                                                    2. 9

                                                                                                                                      As an engineer child of social scientists, I’ve concluded that mental models like that are basically what you get when you take an engineering approach to social systems to its logical conclusion without considering people as, well, people. You end up with very efficient, streamlined, rational systems that place no value upon the people who are crushed in the process. It’s a simple, effective solution to the very complicated problem of human society, and it makes the complicated problem simple by saying “the people on the losing side don’t matter”. You can see this approach working efficiently and effectively all throughout human history, usually in the form of mass graves.

                                                                                                                                      Everything should be made as simple as possible, but no simpler.

                                                                                                                                      1. 3

                                                                                                                                        Because I can’t be sure which comment you’re replying to (AFAIK there’s no “parent” link for comments here), can you please clarify what you mean by “mental models like that”?

                                                                                                                                        1. 4

                                                                                                                                          Sorry, I was talking about mental models such as the ones described by this comment: https://lobste.rs/s/nf3xgg/i_am_leaving_llvm#c_01mpwm . Essentially “we are not going to worry about equity and equality because it is irrelevant to the problem we are actually trying to solve”. Works fine when the problem you are trying to solve is “design a machine that does a particular thing well”, but ignores lots of ugly externalities when it comes down to the social structures and organizations of the people actually doing the design. Like unfettered free-market capitalism, it sounds ideal in theory and that makes it an appealing position. But my observation has been that it works great for the people already powerful enough or lucky enough to be unaffected by those externalities, and does not actually make the world a better place for anyone else.

                                                                                                                                      2. 3

                                                                                                                                        Extremes are rarely good. There should not be an aura of aggressivity around any project.

                                                                                                                                        1. 1

                                                                                                                                          They’re doing literal torch hikes through southern Germany

                                                                                                                                          I have no idea what holding torches might mean in this context. Could you explain, or provide links?

                                                                                                                                          1. 6

                                                                                                                                            It looks like one of those things Nazis ruin for everyone - https://www.theatlantic.com/politics/archive/2017/08/why-they-parade-by-torchlight/537459/. Whether that is intentional on the part of the suckless folks, is not clear to me.

                                                                                                                                            The other top hit I got when googling was a torchlit tourist hike through Partnach Gorge in Garmisch-Partenkirchen. I’ve been to that gorge (not by torchlight) and it’s pretty cool!

                                                                                                                                        1. 2

                                                                                                                                          I think we may be overdue for a tools tag–would that slight generalization satisfy you?

                                                                                                                                          1. 1

                                                                                                                                            Sure looks like this didn’t go over well.

                                                                                                                                          1. 2

                                                                                                                                            “Programming” also covers the programming environnent or does not? A full blown unix programming environment article might be “practices” but a (build) tool alone still fits in the “programming” umbrella which is by default quite large

                                                                                                                                            1. 5

                                                                                                                                              Maybe I just hate the programming tag because it’s so general.

                                                                                                                                            1. 111

                                                                                                                                              Thanks for your efforts!

                                                                                                                                              After four links, a domain can’t have a majority of its stories submitted from a single user.

                                                                                                                                              As a datapoint, I currently can’t submit stories from my domain as I’ve submitted 14 (9 by others). I’m probably biased, but most stories have been reasonably well received so I’d consider that a loss.

                                                                                                                                              1. 47

                                                                                                                                                A simple tweak to this rule: bumping against the ceiling makes new submissions from that domain require mod approval. If posts are consistently well-received, mods can whitelist that (UserName, Domain) pair?

                                                                                                                                                1. 9

                                                                                                                                                  I like this idea! If this is too much moderation overhead, maybe users with sufficiently high karma could see this queue and add approvals?

                                                                                                                                                  1. 12

                                                                                                                                                    Maybe. I dunno. I just threw it out there, but concerns around mod overreach and cabals of power-users are as old as time.

                                                                                                                                                    Tying site privileges to karma creates all sorts of Goodhart’s-law-shaped problems.

                                                                                                                                                    1. 3

                                                                                                                                                      Yeah, but maybe the same trust system that lobsters already has would work here: namely, a mod can delegate this queue to a user they trust? It’s all highly transparent anyway so abuse could be punished?

                                                                                                                                                      1. 2

                                                                                                                                                        A hidden, secondary confidence score that is calculated based on outcomes that are subjectively chosen is where pushcx may be heading with this in due time. Putting a number to it might be a good idea.

                                                                                                                                                  2. 39

                                                                                                                                                    As a datapoint, you are not alone. I wrote:

                                                                                                                                                    in the meantime bumping up against this limit posts a note to moderators so if it goes wrong we’ll see problems

                                                                                                                                                    This definitely went wrong.

                                                                                                                                                    My apologies to those inconvenienced by it, there’s a lot more false positives than I recognized. We’ve had a couple suggestions on how to reduce the error rate like only looking at the last N months or skipping it if any of the domain’s stories have done especially well (better than average or median?). I especially appreciate the folks writing and tweaking queries to try to build up our understanding, and I expect there’s probably some novel angle to separate noise from signal that we’ll think of in the next few days.

                                                                                                                                                    1. 10

                                                                                                                                                      There’s a “homepage” link in the profile. Perhaps the limit could be increased for your declared domain, (possibly, only if it’s unique across users?)

                                                                                                                                                      1. 4

                                                                                                                                                        This is a good idea, but what if the user is submitting from two blogs? For example, their personal blog and the blog of a project (perhaps a project the user contributes to) that the Lobsters community might be interested in.

                                                                                                                                                        1. 8

                                                                                                                                                          We have an authored by checkmark, that might work?

                                                                                                                                                          1. 2

                                                                                                                                                            How many people are doing that? I think it may be acceptable collateral damage.

                                                                                                                                                            1. 1

                                                                                                                                                              Aren’t hats available for that purpose?

                                                                                                                                                              1. 2

                                                                                                                                                                Hats can’t be attached to posts… yet? Also, hats are generally used more for speaking on behalf/with significant involvement for more major projects, less associating sites to users. I suppose it can be changed…

                                                                                                                                                                1. 1

                                                                                                                                                                  To clarify, are you suggesting that hats be used as a signal for increasing the (proposed) limit as to how many times a user can submit stories from a particular domain?

                                                                                                                                                                  1. 2

                                                                                                                                                                    No, but to have people make it clear that they are posting personal or project related. A separate limit per hat would be an idea yes for the post limit.

                                                                                                                                                            2. 2

                                                                                                                                                              Perhaps rate limiting posts rather than an absolute limit (or some combination of trust - whatever that means, account lifespan, etc to generate a score/threshold coupled with rate limits).

                                                                                                                                                            3. 35

                                                                                                                                                              Yes, this rule doesn’t really make sense to me. Users who write good stories will most likely be punished in this category.

                                                                                                                                                              1. 25

                                                                                                                                                                yes, I came to the comment section to ask specifically how to handle posting entries for our own blog posts. I enjoy blogging and this is one of the few places I share my blog posts. Don’t how to handle this now.

                                                                                                                                                                1. 5

                                                                                                                                                                  So, it is mostly me posting my own stories as can be seen in https://lobste.rs/domain/andregarzia.com

                                                                                                                                                                  1. 4

                                                                                                                                                                    Yeah. I don’t blog about stuff as much as I should and lobsters is one of the good signal to noise places I’d wanna share with.

                                                                                                                                                                  2. 17

                                                                                                                                                                    Looking at @arp242 submissions, they look relevant and interesting, so I agree it seems to be a problem with the new algorithm. It will reduce the amount of interesting niche content - precisely what Lobste.rs should be about.

                                                                                                                                                                    I’m probably in the same boat as @arp242 as I submit posts from my domain. One of my submissions is a book announcement with 26 upvotes, and the other five are Elm and Postgres posts and projects, which are neither low-effort nor frequent (this is over two years). I agree with @akkartik’s comment that the timeframe needs to be taken into account too.

                                                                                                                                                                    I was going to suggest that the problem could be addressed by checking whether the user submitted other sites or participated in discussions, with an additional check for community approval in the form of a certain number of upvotes across submissions/comments. However, after looking at @UV’s comment history I see that they would have still gamed that, primarily because it’s still easy to get upvoted low-effort comments here.

                                                                                                                                                                    1. 16

                                                                                                                                                                      Same boat. On the other hand, maybe this will motivate me to start digging through your archives to find interesting things, because I can’t rely on you just posting them here for me ;)

                                                                                                                                                                      1. 11

                                                                                                                                                                        Yeah, it’s a hard choice. I like to think that my own stories, at least as of the past couple of years, are a reasonable fit for this community, and at my current rate of about one post per year I don’t feel like I’m spamming down the site. At the same time, we’ve all seen those account which just post blogspam article after blogspam article from the same domain.

                                                                                                                                                                        Maybe these measures are necessary, but I consider it a good thing that people like yourself, and drew devault, and other people who write in-depth about technology topics they’re genuinely interested in, are able to post their stories here.

                                                                                                                                                                        Besides, this restriction would mostly affect real users who have the community’ best interests at heart, right? If I was a marketing shill and wanted eyeballs I can show content advertising to, I could just create a new account every fourth article, right?

                                                                                                                                                                        1. 8

                                                                                                                                                                          If I was a marketing shill and wanted eyeballs I can show content advertising to, I could just create a new account every fourth article, right?

                                                                                                                                                                          I think we’re actually good in that case! You’d have to invite the alt account, making what you’re doing SUPER obvious. And then we’d bad the entire domain, so you’d never get links from lobsters ever again :D

                                                                                                                                                                          1. 3

                                                                                                                                                                            I sat down at my laptop after work to respond to this because, yes: I was aware of the perverse incentive, but at least it’s pretty darn obvious and it reveals bad intentions. And I was distracted from finishing this comment to investigate and confirm that, yes, this happened.

                                                                                                                                                                            1. 2

                                                                                                                                                                              Why was this user banned? The user submitted 3 things, all of which are relevant and on topic? One of the github links is pretty low quality, but again, not off topic.

                                                                                                                                                                              Or, maybe the things I want to see no longer align with the site…

                                                                                                                                                                              1. 2

                                                                                                                                                                                They were a sockpuppet of vermaden, not a person. I left the three on-topic stories that he submitted as cover for a link promoting his blog.

                                                                                                                                                                                1. 2

                                                                                                                                                                                  Thanks for the explanation!

                                                                                                                                                                                  So, if that’s provably the case that the account was a sock puppet, ban vermaden?

                                                                                                                                                                                  But, how is having multiple accounts any different than asking “joe rando” to post on my behalf, which I did today (it happened to be posted by someone I actually know, but only after I asked)?

                                                                                                                                                                                  I’m going to start following the hashtag on twitter “#promotemeonlobsters” and submit links that appear to be on topic, that don’t appear to be spam to me.

                                                                                                                                                                                  If I get enough people also do this, there will be a wide variety of potential submitters to these stories, making this silly change irrelevant. Additionally, cannot exactly ban submissions found in that stream, since I can plausibly deny I found it there, and not directly from the source by happenstance.

                                                                                                                                                                                  OR, you could stage same domain posters, showing them to a random sampling of users until they reach some upvote threshold (say 3?), at which point everyone can see them. While you’re at it, perhaps this should be the way all posts start out…

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    I thought about banning vermaden for the sockpuppeting, but I don’t read him as a badly intentioned content marketer, I read him as overly enthusiastic about FreeBSD. And if he’s clever enough to find bugs and foolish enough to not realize I’m paying a lot of personal attention to him while he does it, I’d rather let him continue a bit to catch other bugs/misdesigns.

                                                                                                                                                                                    1. 1

                                                                                                                                                                                      I’ve reread your comment multiple times now, am taken aback, and instead of replying how I really feel about it, I’m going to :eyeroll: and leave it be.

                                                                                                                                                                          2. 8

                                                                                                                                                                            Want to second this.

                                                                                                                                                                            It feels like a rule that will punish personal blogs. I’ve been posting stories from my personal blog here before, I’m not sure if there are stories from my blog others posted. I think they match the content people expect here (mostly infosec related) and I don’t think that’s abuse, some of them got well received.

                                                                                                                                                                            If I’d post on medium etc. I wouldn’t have that problem.

                                                                                                                                                                            1. 5

                                                                                                                                                                              It could be time bounded, or tested against multiple time ranges?

                                                                                                                                                                              For instance, user cannot post from a domain if more than half of stories in the last 6 months are from them.

                                                                                                                                                                              Or combine that with the original: a user cannot post a domain if they are more than half of all time posts AND they posted more than half within the last 6 months. That way if you could be the majority of all time, but not the majority of recent posts, or vice versa, and still be allowed to post for a certain domain.

                                                                                                                                                                              And “the last 6 months” could be 3 months, could be 1 year, or what-have-you.

                                                                                                                                                                              1. 3

                                                                                                                                                                                I agree. The four link thing is kinda messed up. I write essays every couple of weeks or so, all about making better tech. I’ve spent some time making sure there’s no ads, no email lists, no sales at all on these links. I can’t make them any more innocuous, and I use the comments and feedback from here to learn more and (hopefully) write better stuff.

                                                                                                                                                                                It doesn’t make sense that I can’t keep doing this. Perhaps the four link rule would work better when the domains were already high traffic sites? You don’t wanna kill the mom and pop grocery stores simply because you don’t like the new WalMart moving into town.

                                                                                                                                                                              1. 17

                                                                                                                                                                                Spitballing a couple ideas here.

                                                                                                                                                                                On the topic of the 70 day new-user timer:I think this will just make the spammers much more difficult to notice as they will create dormant accounts waiting for the timer to expire while maybe posting a low-effort comment here and there. Possibly even automating it by copying or generating content based on previous comments on lobste.rs. I don’t see a direct solution to this, but it’s worth to keep in mind how their approach will change with this.

                                                                                                                                                                                Regarding the new-user limitations in general: How about having two different types of invites you can send out? One regular invite as it is now, and one for people you trust? Sending the trusted invite would mean you personally take responsibility for actions taken by the person you invite (for a reasonable time frame that is) and their limitations are relaxed somewhat. This means you still can extend invites to people you don’t trust all that much, and they would end up having to display their trustworthiness, or you can shortcut that mechanism and allow someone you already trust onto the platform.

                                                                                                                                                                                1. 12

                                                                                                                                                                                  Why would you invite somebody you don’t trust? Two levels of that and we’re back to where we started.

                                                                                                                                                                                  I’m not totally sold on a time-based “newness” metric. Something more informed by usage would be good–are they submitting new content, are they actively commenting, are they helping suggest tags and whatnot. And sure, those are all gameable, but if we can trick the growth droids into performing vital community service why not?

                                                                                                                                                                                  1. 31

                                                                                                                                                                                    I was invited by @flyingfisch because I asked on IRC. He invited me in good faith but doesn’t trust me as a personal friend. If I was banned he might get told “hey be more careful” but that’s about it. There are a couple of people I invited like that. I also invited @rwhaling because he’s someone I trust. If he got banned, I’d feel obligated to apologize to the lobsters community, because I personally “vouched” for him and was wrong.

                                                                                                                                                                                    That’s how I see this. I’d come in as a “standard” newbie and would have all the restrictions at first. @rwhaling would come in as a “vouched-for” user and would have fewer restrictions. But if he got banned, I’d be suspended or put on probation or something.

                                                                                                                                                                                    1. 6

                                                                                                                                                                                      Nice explanation. As a newbie who’s not enitrely sure how the whole community “breathes”, I’m constantly afraid of posting something so bad that my inviter has to feel bad about me. And they only gave me invite because I saw on Mastodon that they are a user here, we have low interraction otherwise.

                                                                                                                                                                                      1. 5

                                                                                                                                                                                        Thanks for your explanation!

                                                                                                                                                                                        But if he got banned, I’d be suspended or put on probation or something.

                                                                                                                                                                                        I like this approach for taking responsibility for downtree users.

                                                                                                                                                                                        1. 5

                                                                                                                                                                                          Suspend future invitations for some time.

                                                                                                                                                                                        2. 1

                                                                                                                                                                                          I think leaning into the social network/web of trust is a good idea. I think it may be useful to express beliefs and confidences, about users you do not invite.

                                                                                                                                                                                          What reward you get to balance the risk ventured, I don’t know. Maybe just the knowledge that you’re helping the anti-spam network.

                                                                                                                                                                                          1. 1

                                                                                                                                                                                            This is exactly the mechanics I was thinking about. The punishments and rules of it would need to be hashed out, but you are spot on with the idea.

                                                                                                                                                                                        3. 9

                                                                                                                                                                                          I think this will just make the spammers much more difficult to notice as they will create dormant accounts waiting for the timer to expire while maybe posting a low-effort comment here and there.

                                                                                                                                                                                          This is certainly existentially possible, but not actually true in my experience. One kind of abuse I deal with is fraudulent activity in my billing system. One surprising aspect of these behaviors is how impatient the people engaging in them are. Instead of waiting for the ‘right’ opportunity, they go to where there is an immediate ‘return’ on their effort. One explanation for this is that it reduces the evidence of the behavior, lowering legal risk. I think part of it is high time preference, however.

                                                                                                                                                                                          It is true that there are innovations in parasitism (e.g., spamming), just as there are innovations in productive and positive behaviors. What happened here was “innovative” in the sense that we had not seen it before. Since it provided an unearned benefit, the behavior was repeated until discovered and suppressed. That will happen again, perhaps even in the manner you articulate but probably in a more novel way–but that is going to be true every time we suppress an unwelcome behavior. Incremental suppression increases the cost of imposing on us–requiring more effort for the same reward is a feature, regardless of whether it fully eliminates the unproductive behavior or not.

                                                                                                                                                                                          1. 7

                                                                                                                                                                                            This also happens with SMTP servers, interestingly. One of postfix’s most basic spam-prevention settings is just waiting a small amount of time at the beginning of each SMTP session and canning it if the client talks first. Apparently, the server is meant to talk first, but most spambots are so impatient (because they have to spam as much as possible before they get blacklisted) that they send their HELO before the server has sent them anything.

                                                                                                                                                                                          2. 4

                                                                                                                                                                                            I wonder if it would be possible to take the max between 70 days and an arbitrary karma count (maybe the median user karma level?). New accounts that genuinely want to join the community and actively participate shouldn’t be tagged as potential spammers for over 2 months.

                                                                                                                                                                                            When I was invited 5 years ago, the person sending the invite was responsible for the new users that they invited and could lose their account/run into trouble if they abused the invite feature. It is my understanding that that was the main reason for showing the invite tree back then. Does anyone know if that policy has changed or if I just misremember the “good old times”?

                                                                                                                                                                                            1. 5

                                                                                                                                                                                              In practice “upstream” users have only been banned for downstream user’s indiscretions once or twice. It basically doesn’t happen.

                                                                                                                                                                                              1. 1

                                                                                                                                                                                                Off the top of my head, I only know of one user banned because of behavior by someone they invited, and they wrote the code to disable invites, so I don’t think there have been any since then.

                                                                                                                                                                                            2. 2

                                                                                                                                                                                              Expect owners of older accounts that aren’t used that much to start getting cash offers for them…

                                                                                                                                                                                            1. 8

                                                                                                                                                                                              Does someone have a proper doc on this I could replace this thin blog post with?

                                                                                                                                                                                              1. 5

                                                                                                                                                                                                There is the document on homed itself here: https://systemd.io/HOME_DIRECTORY/

                                                                                                                                                                                                The merge request is here, merged 10 days ago: https://github.com/systemd/systemd/commit/4119d1e60a111bacca359b61b5bc3dae29932b67

                                                                                                                                                                                                1. 4

                                                                                                                                                                                                  My gut feeling tells me it’s mostly due to the fact that a hash value of an int is the int itself, so there’s no time wasted on hashing.

                                                                                                                                                                                                  Oh wow I hope not. Is this actually true in CPython?

                                                                                                                                                                                                  1. 6

                                                                                                                                                                                                    This is the most sensible implementation as you want to avoid collisions in a hash table. It isn’t supposed to bear any cryptographic properties if that’s your concern. Here’s more: https://github.com/python/cpython/blob/master/Objects/dictobject.c#L134

                                                                                                                                                                                                    1. 5

                                                                                                                                                                                                      It’s not the most sensible implementation, because simple key patterns cause collisions that never resolve, even when resizing the hashtable. The comment you linked specifically mentions this pathology, and the numerous ways it destroys performance.

                                                                                                                                                                                                      The rest of the comment describes how CPython mitigates the issue by adding weak integer hashing to its collision probing. At first I thought integer keys were never hashed at any point, hence my surprise.

                                                                                                                                                                                                      From the comments it sounds like sequential integer dict keys are somehow common in Python, which I don’t understand. But I don’t write much Python.

                                                                                                                                                                                                      1. 6

                                                                                                                                                                                                        From the comments it sounds like sequential integer dict keys are somehow common in Python, which I don’t understand. But I don’t write much Python.

                                                                                                                                                                                                        While you can have a dict with keys of any hashable type – and a single dict may have keys of many types – the most common case, so overwhelmingly more common that it’s almost not even worth thinking about other cases, is a dict whose keys are all strings. This is because, sooner or later, basically everything in Python is backed by a dict. Every namespace is backed by a dict with string keys (the names defined in that namespace). Every object is backed by a dict with string keys (the names of the object’s attributes and methods). Keyword arguments to functions/methods? Yup, dict. In comparisons of languages by their “one big idea”, Python is sometimes described as having its big idea be “what if everything was a string-keyed hash table”?

                                                                                                                                                                                                        Anyway. This is so common that Python goes out of its way to have special-case optimized implementations for the case of a dict whose keys are all strings (and for what it’s worth, in CPython as of Python 3.4, str is hashed using SipHash-2-4).

                                                                                                                                                                                                        As to hashing of numeric types, it’s a bit more complicated than “ints hash to themselves”. Here’s what the Python documentation has to say. For the specific case of int, you can think of it as reducing to hash(n) == hash(n % sys.hash_info.modulus), where in CPython sys.hash_info.modulus is 2^61 - 1 on systems with 64-bit long and 2^31 - 1 on systems with 32-bit long.

                                                                                                                                                                                                        While I don’t have a way of being certain, I suspect the linked comment’s note that the hashing of int is “important” has to do with the importance of real-world int key values being unlikely to collide with the hashes of other common real-world key types.

                                                                                                                                                                                                        1. 1

                                                                                                                                                                                                          In comparisons of languages by their “one big idea”, Python is sometimes described as having its big idea be “what if everything was a string-keyed hash table”?

                                                                                                                                                                                                          I’ve always thought of PHP’s “one big idea” as “What if everything is an array” where array means PHP’s strange half-dict half-list interface that funnily enough Python is now one small step closer to.

                                                                                                                                                                                                      2. 1

                                                                                                                                                                                                        Avoiding collisions isn’t as important as using up a larger % of the spots before you need allocate and move things, I believe.

                                                                                                                                                                                                        1. 3

                                                                                                                                                                                                          Aren’t those the same thing? Less collisions implies you can go longer without expanding.

                                                                                                                                                                                                          1. 1

                                                                                                                                                                                                            It depends on the exact implementation, but in my understanding, not exactly; you also want a good distribution between your buckets, even if there are patterns / non-random distributions in the actual encountered keys. It might waste space rather than time, but it’s still not great.

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              Python’s hash table isn’t implemented as an array-of-buckets. It’s a single contiguous array into which you insert a new element at the position determined by the hash of its key, and if that position is occupied you try the next one in a pseudo random order. Same with lookups: you try entries in succession until you find the one that equals (it’s usually the first one). And this is why the number of free spots and the probability of collisions are directly related.

                                                                                                                                                                                                      3. 2

                                                                                                                                                                                                        it is:

                                                                                                                                                                                                        Python 3.7.6 (default, Dec 21 2019, 11:56:31)
                                                                                                                                                                                                        [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
                                                                                                                                                                                                        Type "help", "copyright", "credits" or "license" for more information.
                                                                                                                                                                                                        >>> hash(2)
                                                                                                                                                                                                        2
                                                                                                                                                                                                        >>> hash(37)
                                                                                                                                                                                                        37
                                                                                                                                                                                                        >>> hash(892474)
                                                                                                                                                                                                        892474
                                                                                                                                                                                                        
                                                                                                                                                                                                        1. 8

                                                                                                                                                                                                          Almost! hash(-1) returns -2.

                                                                                                                                                                                                          Python 3.8.1 (default, Jan  8 2020, 23:09:20)
                                                                                                                                                                                                          [GCC 9.2.0] on linux
                                                                                                                                                                                                          Type "help", "copyright", "credits" or "license" for more information.
                                                                                                                                                                                                          >>> hash(-1)
                                                                                                                                                                                                          -2
                                                                                                                                                                                                          >>> hash(-2)
                                                                                                                                                                                                          -2
                                                                                                                                                                                                          >>> hash(-3)
                                                                                                                                                                                                          -3
                                                                                                                                                                                                          
                                                                                                                                                                                                          1. 4

                                                                                                                                                                                                            wat

                                                                                                                                                                                                            do you happen to know why?

                                                                                                                                                                                                            1. 7

                                                                                                                                                                                                              Ah, it’s because the C API function uses -1 as an error code. It goes deeper than that too:

                                                                                                                                                                                                              In [1]: class yolo:
                                                                                                                                                                                                                 ...:     def __hash__(self):
                                                                                                                                                                                                                 ...:         return -1
                                                                                                                                                                                                                 ...:
                                                                                                                                                                                                              
                                                                                                                                                                                                              In [2]: y = yolo()
                                                                                                                                                                                                              
                                                                                                                                                                                                              In [3]: hash(y)
                                                                                                                                                                                                              Out[3]: -2
                                                                                                                                                                                                              
                                                                                                                                                                                                          2. 2

                                                                                                                                                                                                            I’ve heard that this is a somewhat common way to implement hashing for ints, but I don’t understand why it’s a good idea. Isn’t hash collisions terrible for hash tables? And isn’t a somewhat common key pattern “some number with some low bits masked”? And wouldn’t that be a pathological case for a hash table which grows with a factor of 2?

                                                                                                                                                                                                            Are hash table implementations which does hash(x) = x somehow better at handling collisions than most hash tables, or do they just hope that the ints people want to put in their tables have high entropy in the lower bits?

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              IIRC there is some sort of random salt added to it and it goes through some internal hash for the actual hash table, since there was a DoS attack by abusing worst case scenario over HTTP requests.

                                                                                                                                                                                                          3. 2

                                                                                                                                                                                                            Why would that be a problem?

                                                                                                                                                                                                          1. 7

                                                                                                                                                                                                            Was the link supposed to lead to the submission for last year’s competition? I joined our leaderboard, under the name grxnola.

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              Shoot! I pinged pushcx to edit the URL.

                                                                                                                                                                                                            1. 9

                                                                                                                                                                                                              With Python 3 being out for so long and the Python 2 deadline being delayed I think it is somewhat embarassing Mercurial still doesn’t use it as default. Sure, 3rd party plugins are not up to date, but the longer they wait with the migration, the longer Python 2 will need to be kept alive.

                                                                                                                                                                                                              1. 13

                                                                                                                                                                                                                Python 3 has been especially difficult for Mercurial because Mercurial does not use encodings. It’s an application to store your bytes and your bytes are just bytes. You don’t want Mercurial (or git, for that matter) to start having opinions about how to interpret the bytes you give it. All the data that hg deals with – all of it – is just bytes.

                                                                                                                                                                                                                The Python 2 deprecation deadline was mostly extended by 5 years for Mercurial. A bunch of us (not just Mercurial, but probably we were the loudest ones) crowded around Guido at Pycon 2014 and told him about how difficult Python 3 was for Mercurial. In response, Guido later announced that day that we would have 5 more years. The bare minimum we needed was PEP 461, which wasn’t available back in 2014, and once that was done, the rest was just a lot of hard work.

                                                                                                                                                                                                                I am not at all embarrassed by the gargantuan effort that my fellow Mercurial maintainers have made to port Mercurial to Python 3. Quite the contrary, I’m proud of them and have gotten a little resentful that the Python 3 upgrade path has not been nearly as easy as its proponents claim.

                                                                                                                                                                                                                1. 6

                                                                                                                                                                                                                  Oil is the same way – it doesn’t use Python’s unicode object. it uses bytes, since libc calls all take and return bytes (e.g. execve, chdir, listing directory entries, etc.)

                                                                                                                                                                                                                  http://www.oilshell.org/blog/2018/03/04.html#faq

                                                                                                                                                                                                                  Right now the Python 2 interpreter is bundled with Oil. I removed at least half of the interpreter, which I’m not using [1] (e.g. the whole front end, unicodeobject.c, complexobject.c, etc.)

                                                                                                                                                                                                                  Eventually I want to get rid of the dependence on the interpreter. But I think this was a decent way to make progress quickly and have a working program the whole time.

                                                                                                                                                                                                                  Since hg is already a mature and working program, I imagine bundling the interpreter might also have been a worthwhile strategy.

                                                                                                                                                                                                                  It would also largely fix the startup time issue (which I think got worse in Python 3) since you’re not dependent on the system PYTHONPATH and you can modify the startup sequence with some #ifdefs.

                                                                                                                                                                                                                  People always talk about maintenance of Python 2 but I don’t really think it’s a problem. It’s a very mature and understandable codebase. I’ve made dozens of small, localized modifications to it.

                                                                                                                                                                                                                  And if Python 2 has a security issue in say the tempfile module, well it doesn’t affect a program that doesn’t use that module.

                                                                                                                                                                                                                  [1] http://www.oilshell.org/blog/2018/11/15.html

                                                                                                                                                                                                                  1. 6

                                                                                                                                                                                                                    We’re actually vaguely planning to bundle Python with Mercurial via PyOxidizer - it should be a one-file binary at that point.

                                                                                                                                                                                                                    I’m sure Greg would be interested in Oil doing it too, but I think it can’t be done on Python 2 for a variety of reasons.

                                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                                      People always talk about maintenance of Python 2 but I don’t really think it’s a problem. It’s a very mature and understandable codebase. I’ve made dozens of small, localized modifications to it.

                                                                                                                                                                                                                      So you’ve made dozens of small modifications to it (python) and are now carrying those changes, and you don’t see a problem with many people doing the same thing? It seems like you’ll quickly end up in a situation where one implementation of python2 used by/included in one program is not the same as another used by/included in another, and that is a bigger maintenance problem.

                                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                                        Why is that a maintenance problem? There can be N different forks of Python 2, e.g. one used for hg, one used for Oil, and I don’t see the problem. Each team maintains it separately.

                                                                                                                                                                                                                        Python 2 is a stable, well-tested, and well-understood codebase.

                                                                                                                                                                                                                        It’s somewhat analogous to there being N forks of the BSD kernel. In that case I’d say it’s also a feature and not a bug.

                                                                                                                                                                                                                        Although you could argue why they don’t combine their efforts (though I don’t think it makes sense to argue). Either way, that argument doesn’t make sense for hg or Oil or dozens of other programs that are written in Python and may want to make small, local, backward incompatible modifications.

                                                                                                                                                                                                                        1. 1

                                                                                                                                                                                                                          The problem is that you have a bunch of different teams maintaining essentially (but not quite) the same thing in their own gardens. So when a vulnerability is found, you have to rely on them all to 1) notice it then 2) fix/deploy. And users now have N number of “python 2” installs on their one system that are in all sorts of different states. How do you know if you have something using a broken “version” of python 2? When a CVE is posted, how can you possibly tell if you’re affected in all the different “versions” of python 2 on your system?

                                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                                            On balance, I think using Python 2.7 and statically linking it reduces the number of vulnerabilities. (Although it’s not ideal for other non-security reasons, which is why it’s not a permanent solution).

                                                                                                                                                                                                                            If you rewrite the functionality from scratch rather than reusing well-tested pieces of code, you’d have as many or more vulnerabilities in your own code. Unless of course there is something you’re doing that the Python team isn’t (possible but not that likely).

                                                                                                                                                                                                                            I’d say Python 2.7 more solid than the average piece of code simply because it’s undergone a lot of testing and usage. There have been a lot of eyeballs on that code.

                                                                                                                                                                                                                            And as mentioned, the vulnerabilities in stdlib modules that Oil don’t use don’t matter because they’re not even in the binary (e.g. I see CVEs in XMLRPCServer, which has nothing to with Oil). On top of that, I also remove around 150K lines of other interpreter code, leaving somewhere around 100K. That’s more than I would like, but it’s not a problem to maintain.


                                                                                                                                                                                                                            On a related note, I think there is some cargo culting of the “CVE treadmill”. Newer code isn’t always better. If the development team isn’t careful about security, you can patch one CVE and introduce another one. In fact I think that’s pretty common because software tends to get larger over time.

                                                                                                                                                                                                                            I’d rather move to a model where there is actual reasoning about security from first principles rather than “I heard about this CVE so I need to update all the places where it occurs, and now I did, so the problem is gone”. It’s kind of where the industry is now, but it doesn’t reflect the underlying reality.

                                                                                                                                                                                                                    2. 2

                                                                                                                                                                                                                      bare minimum we needed was PEP 461

                                                                                                                                                                                                                      Can you expand on that? That request looks just like some convenience feature that could have just been a normal method.

                                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                                        It’s because mercurial used % formatting on python2 strings heavily to encode the mercurial wire protocol. Before pep 461, % formatting on bytestrings was not allowed, so they would have needed to carefully rewrite the wire protocol code in a way that would have left the code less readable.

                                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                                          Isn’t that pretty much a basic refactoring operation in any typed language?

                                                                                                                                                                                                                          1. 7

                                                                                                                                                                                                                            If you take a stroll through the gargantuan effort of hundreds of commits I linked above, I think you’ll see that it wasn’t a basic refactoring operation.

                                                                                                                                                                                                                            1. 4

                                                                                                                                                                                                                              Not sure what you mean, regardless it doesn’t matter much for Mercurial because it’s not written in a typed language.

                                                                                                                                                                                                                              Also I think the aspect of rewriting the code for no gain but leaving the code less readable is a big deal, readability counts.

                                                                                                                                                                                                                        2. 1

                                                                                                                                                                                                                          It’s an application to store your bytes and your bytes are just bytes.

                                                                                                                                                                                                                          Python 3 has a “just bytes” type. One that’s actually far better than Python 2’s, because Python 2’s was overloaded by also needing to be the default string object. For example, a while ago I went back and did the first couple sets of cryptopals challenges again using Python 3 instead of 2, and it was night-and-day how much nicer Python 3 was for working with “just bytes”.

                                                                                                                                                                                                                          And the fact that Mercurial was apparently utterly blocked by not being able to use % on bytes objects indicates Mercurial was not treating these bytes as “just bytes” – the % operator is there to let you do string formatting operations with printf()-style syntax, which only makes sense if the bytes are in fact being treated as strings (which in turn means you need to “use encodings”, because formatting operations on byte sequences of unknown encoding are a ticking time bomb in the code).

                                                                                                                                                                                                                          1. 6

                                                                                                                                                                                                                            Please read PEP 461. Its use cases are explained there and apply to Mercurial.

                                                                                                                                                                                                                            1. 4

                                                                                                                                                                                                                              I have read it. I’m not disagreeing that the Mercurial team says PEP 461 makes the their lives easier. I’m disagreeing that use of % on str in Python 2 was a “just bytes” operation.

                                                                                                                                                                                                                              Even the rationale in the PEP, and the discussion on python-dev when the PEP was being drafted, acknowledged this – the whole purpose for this is to allow treating some bytes as actually being encoded text, and manipulate them in ways that only make sense for encoded text. Which in turn carries an assumption about what encoding was used to produce the bytes, etc.

                                                                                                                                                                                                                              Which means that all of your insistence on “just bytes”, “no encodings”, “no interpretation”, and so on in your original comment was, to be charitable, a red herring. Mercurial didn’t need % formatting for “just bytes” operations; it needed it for operations on bytes that Mercurial assumes are ASCII-encoded text.

                                                                                                                                                                                                                              I don’t doubt it’s perceived as easier for Mercurial to continue doing things that way. I do doubt whether it’s the right approach to wire protocols which consist of mixed encoded text and “just bytes”.

                                                                                                                                                                                                                        3. 4

                                                                                                                                                                                                                          Actually with the latest release (Mercurial 5.2), Python3 is the default.

                                                                                                                                                                                                                          1. 1

                                                                                                                                                                                                                            …except on Windows, unless I missed something, correct?

                                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                                              Yep. Windows has a little ways to go yet. :(