Threads for goalieca

  1. 2

    It would be nice to have a standard API or html element or something to make selection uniform across all sites. We could integrate it into the browser settings.

    1. 3

      A request header perhaps…

      The problem is, the sites with the dodgy banners want you to accept their tracking and cookies. It is not in their interest to make opt-out easier.

      1. 4

        Let’s call it Do-Not-Track, but write it as DNT to make it shorter.

    1. 2

      Now all that’s missing is enteprise tooling :)

      1. 33

        At a previous job, one of the other developers and I used to have a sort of competition on who could remove the most likes of code per week, while doing normal work.

        I remember winning by 1.2 million lines one week when I found a second, unused, copy of a service embedded in another service’s repo.

        I do kind of miss that friendly competition

        1. 8

          I do kind of miss that friendly competition

          Now I’m imagining that the competition turned unfriendly and your coworker began to delete production services in order to beat your line count…

          1. 11

            If no one complained, give this guy a medal!

            1. 7

              This is also known as the “scream test” when used to measure whether or not a given thing was in use at all. ❤️

              1. 2

                Sooooo … how long do you wait before declaring an absence of screams?

                (Quietly considers EOFY processes that run once per year …)

            2. 2

              If production kept operating correctly…why not? :)

              1. 6

                Delete your backups system and production will keep operating correctly.

                1. 6

                  If that doesn’t raise any alerts, then it wasn’t operating correctly in the first place.

                  1. 12

                    Delete the alerts

            3. 6

              I once worked in an office with a similar culture. The competition ended when someone found a header file containing a dictionary as a C array in some test code. They replaced it with a short function that reads the dictionary that comes with the distro, and removed 700,000 lines

              1. 6

                My favourite kinds of diff to review

                1. More features, less code
                2. No functionality change, less code
                3. More features, same amount of code

                One of the reasons that I hate C so much is that it’s incredibly rare to see diffs of any of these kinds in a language that doesn’t provide rich types. It’s common in C++ to review diffs that move duplicated code into a template or lambda (you may end up with the same amount of object code at the end, but you have less source to review and bugs are easier to fix).

                1. 1

                  Hehe, we had that at a previous job as well, but the highest score was in the thousands, 1.2 million is awesome, hahahaha

                1. 1

                  I made this as a personal challenge about 15 years ago (before c++11) after i learned Haskell. I got pretty far along before i had to cave in because some data arrived in two separate lists but were related as if they were a single list of tuples. That’s when things became less elegant to mess around with. Also, native schemes like python enumerate() or haskell’s zipWith \x [1..] are elegant ways to introduce counters.

                  1. 3

                    I have a Raspberry Pi 4 in need of a project, so I am going to turn it into an access point (as per this post: https://willhaley.com/blog/raspberry-pi-wifi-ethernet-bridge/). I have a spot in the house that is going to be hard to run Ethernet cable to, so this is a good use for it, although overkill.

                    1. 2

                      I did the opposite somewhat. I put my wifi router as the access point and my rpi as the internet router.

                    1. 16

                      There’s a lot of good stuff in here that we all think everyone knows and we say to each other in the pub but we don’t really say out loud to the people that need to hear it.

                      The main one that comes to mind is about mobility. They said something like “if I get fired I’ll have a new job in two weeks.” The tech folks that don’t know this is true need to learn it. More importantly: the people who manage tech people need to learn it.

                      1. 22

                        if I get fired I’ll have a new job in two weeks.

                        This has never been true for me. Job hunting has always been a relentless slog.

                        1. 12

                          Imma guess it depends on where you are. Silicon Valley, Seattle, NYC, London, you can basically put your desk stuff in a box, and throw it out a window and have it land in another tech company’s lobby.

                          Other places, not so much.

                          1. 9

                            I agree living in a tech hub makes finding a job way easier, but I jump to temper the hyperbole just a bit. I know that I personally felt a lot of self-hatred when I tried to change jobs and it took months of applications and references and interviews to actually get one, even living in a tech hub.

                            1. 6

                              Technology stacks don’t really matter because there are like 15 basic patterns of software engineering in my field that apply. I work in data so it’s not going to be the same as webdev or embedded.

                              It depends on what you do. The author is a database specialist, so of course they’re going to claim that SQL is the ultimate language and that jobs are plentiful. I’m an SRE, so my career path requires me to pick specific backend-ready languages to learn. I have several great memories of failed interviews because I didn’t have precisely the right tools under the belt:

                              • I worked on a Free Software library in Python along with other folks. They invited me to interview at their employer. Their employer offered me a position writing Lua for production backends. To this day, I still think that this was a bait-and-switch.
                              • I interviewed at a local startup that was personally significant in my life. I had known that it wasn’t a good fit. Their champion had just quit and left behind a frontend written with the trendiest JS libraries, locking their main product into a rigid unmaintainable monolith. I didn’t know the exact combination of five libraries that they had used.
                              • I interviewed at a multinational group for a position handling Kubernetes. I gathered that they had their own in-house monitoring instead of Prometheus, in-house authentication, etc. They also had a clothing line, and I’m still not sure whether I was turned down because I didn’t already know their in-house tools or because I wasn’t wearing their clothes.
                              1. 3

                                They also had a clothing line, and I’m still not sure whether I was turned down because I didn’t already know their in-house tools or because I wasn’t wearing their clothes.

                                Seems like a blessing in disguise if it was the clothes.

                              2. 3

                                I have this problem and I’m in a tech hub. Most of my coworkers and technical friends are in different countries I can’t legally work in, so I rarely get interviews through networking. Interviewing is also not smooth sailing afterwards.

                              3. 5

                                This has never been true for me. Job hunting has always been a relentless slog.

                                Same here, I also live in a city with many startups, but companies I actually want to work for, which do things I think are worthwhile, are very rare.

                              4. 7

                                There’s a lot of good stuff in here that we all think everyone knows and we say to each other in the pub but we don’t really say out loud to the people that need to hear it.

                                Interesting that you say that in the context of modern IT. It has been so with many things since ancient time.

                                https://en.wikipedia.org/wiki/In_vino_veritas

                                Perhaps the traditional after-work Friday beer plays a more important role in one’s career than most people think. Wisdom is valuable and not available ons course you can sign up to.

                                1. 1

                                  Wisdom is valuable and not available ons course you can sign up to.

                                  Which is ironic given wisdom is often what they’re being sold as providing.

                                2. 5

                                  The main one that comes to mind is about mobility. They said something like “if I get fired I’ll have a new job in two weeks.” The tech folks that don’t know this is true need to learn it. More importantly: the people who manage tech people need to learn it.

                                  Retention is a big problem. It can take up to a year to ramp up even a senior person to be fully productive on a complicated legacy code base. Take care of your employees and make sure they are paid a fair wage and not under the pressure cooker of bad management who thinks yelling fixes problems.

                                  1. 2

                                    That’s probably why the OP says their salary went up 50% while their responsibilities reduced by 50%. Onboarding.

                                1. 3

                                  I do some c and c++ for my dayjob. If you forego inheritance, modern C++ isn’t so bad. Just stick to std:: types and functions wherever possible and you’ll be reasonably safe, expressive, and performant. The standard library is improving but simple things like parsing integers and tokenizing strings still really sucks and is very complicated with string views.

                                  1. 26

                                    (Disclaimer: I’m a Microsoft employee.)

                                    The way to think about this is there are multiple reasons for a BSOD.

                                    1. Windows has a bug.
                                    2. A driver has a bug. Note drivers are in the same protection domain so can modify almost any system memory.
                                    3. Hardware has a bug in that it corrupts memory.

                                    The reason that people disagree over stability is because (2) & (3) are much more likely than (1), so crashes can plague particular configurations while leaving others completely unaffected. It’s very hard for mortals to pinpoint a driver or hardware bug, so all all users see is the result, not the cause.

                                    The part that always frustrates me a little is people who overclock devices, causing hardware errors, and blame the result on Windows. It’s not that Windows is worse than any other kernel, it’s that the people who overclock hardware all seem to run Windows.

                                    1. 12

                                      My impression is that the Windows kernel is really top notch these days (as opposed, to say, the drivers, display manager, etc, etc).

                                      1. 4

                                        I agree. The one thing I think Windows must improve is in its modularity and letting the user chose which applications and services to be installed.

                                        There are too many services and features I’d like to be able to remove (Or better, chose not to install). There was a talking about Windows Mini Kernel, I want that. I want efficiency.

                                        1. 4

                                          Have you tried Windows Embedded? Server Core? WinPE?

                                          The guts of Windows is fairly modular and composable. The issue is that each of those services are providing something, so removing them will affect applications or scenarios in ways that may not be obvious. The monolithic nature of Windows is really a result of trying to ensure that programs work, and work the same way, on each machine.

                                          Personally I do a lot of command line development, so I thought Server Core would be an interesting option. Here’s what happened:

                                          1. The Visual Studio installer is an Electron application, so it failed because a DirectX DLL wasn’t present;
                                          2. I put the relevant DLL there, and the installer launched with a lot of rendering glitches since it’s a pure-GDI non-composited environment, but I got things to install;
                                          3. Programs using common controls don’t render correctly, which isn’t a big deal for servers, but makes certain things like GFlags to be nigh incomprehensible;
                                          4. …but the end result was the programs I was writing behave differently where appcompat shims and services aren’t running. In a lot of ways I don’t miss them, but the consequence is I can’t run my program in this environment and assume it works the same way in a real environment, so it wasn’t useful for development work.
                                          1. 2

                                            It sounds like a mess. Maybe I should take back my words :-).

                                            One of the issues ow Windows is the luggage it carries. It is time you put all pre historic compatibility under a VM and be done with it.

                                            Moreover, I het what you say and still I’d be happy to have user choices to what to install. Windows is bloated. 30 GB for OS is too much. The RAM consumption is too much. Performance are getting better and hopefully one day we’ll a File System as fast as Linux and the margin will be negligible.

                                          2. 3

                                            I’d love to pay for a gaming build of Windows that only includes necessary components and presumes that I’m competent enough to steward maintenance of my own machine.

                                            1. 2

                                              If you want a gaming build of Windows, you can buy that. It even comes bundled with a computer optimised for running it.

                                        2. 5

                                          I worked as a repair tech in a computer shop for about three years; this was over ten years ago so most of my experience is with XP, Vista, and 7. In this time I saw a lot of BSODs.

                                          In my experience the overwhelming majority of BSODs are caused by faulty hardware or driver bugs. For example the Dutch version of AT&&T (KPN) handed out these Realtek wireless dongles for a while, but after some update in XP they caused frequent BSODs. I’m going to guess this was Realtek’s fault and not Microsoft’s, and it just happened to work prior to this update (they never released an update to fix this. They also never made Vista/7 drivers). Plenty of customers were quick to blame Microsoft for this though, in some cases even after I explained all of this to them they still blamed Microsoft.

                                          By far the most common problem though was just faulty memory. By my rough estimate it caused at least half of all problems, if not more, during this time. The rest were a combination of other hardware faults (mainboard, hard drive, etc.) or bad (often third-party) drivers.

                                          No doubt BSODs happen due to Windows bugs, but it’s a lot less often than some people think. The biggest issue was actually the lack of tooling. Windows leaves small “minidump” core dumps, but actually reading them and getting an error isn’t easy. I actually wrote a Python script to read them all and list all reasons in a Tkinter window, and this usually gave you a pretty good idea what the problem was.

                                          1. 3

                                            Even if i despise Windows nowadays, i agree with you and BSOD stability isn’t a problem nowadays anmore. There are a lot of problems, but kernel stability ain’t one

                                            1. 2

                                              I think it is fair that windows maintains some criticism. A micro kernel would not suffer a systemic failure from a buggy audio driver for instance. Linux is also another insane system where driver code for dozens of architectures are effectively maintained on a budget but i rarely see any crashes on my commodity development box that corporate procured. My dell laptops running win7 and win10 have all crashed frequently.

                                              1. 8

                                                I think some of the stability that you see on Linux is that the drivers are upstreamed, and so face the same discipline as the rest of the kernel, whereas Windows drivers are often vendor-supplied, and potentially very dodgy. You can easily crash Linux with out-of-kernel-tree drivers, but there are only a few of those that are in common use.

                                                1. 1

                                                  Much of the audio stack in Windows runs in userspace. You can often fix audio driver crashes by restarting the relevant services. The troubleshooting wizard does this for you.

                                                  Linux and Windows are both moving to more device drivers in userspace. CUSE on Linux, for example, and Windows also has a framework for userspace USB drivers. Most GPU drivers are almost entirely userspace, for performance reasons: the devices support SR-IOV or similar and allow the kernel to just map a virtual context directly into the userspace address space, so you don’t need a system call to communicate with the device.

                                                2. 1

                                                  On the one hand it’s a bit unfair to blame current windows for earlier disgressions but it is what it is.

                                                  Regarding your point 3) - I’ve had it SO often that a machine in the 98-XP days would crash on Windows and run for a week on Linux, so I don’t really buy that point. Hardware defects in my experience are quite reproducible “every time I start a game -> graphics card”, every time it runs for longer than a day -> RAM, etc.pp. Nothing of “it crashes randomly every odd day” has ever been a hardware defect for me (except maybe ram, and that is sooo rare).

                                                  I don’t think I have claimed Windows is unstable since I’ve been using 7 or 10 (and 2000 and XP were okish). But 98 (non-SE), Me, Vista, 95a and 95b were hot garbage.

                                                1. 22

                                                  The fact that we even need to worry about sandboxing for looking at glorified text documents is embarrassing.

                                                  1. 27

                                                    Your PDF reader also ought to be sandboxed; malicious PDF documents have used to hack people.

                                                    Ideally, your ODT reader also ought to be sandboxed. There have been RCE bugs in LibreOffice where malicious documents could exploit people.

                                                    Reading untrusted user input is hard. Hell, even just font parsing is fraught with issues; Windows’s in-kernel font parser was a frequent target of bad actors, so Microsoft sandboxed it.

                                                    Sandboxing is almost always a good idea for any software which has to parse untrusted user input. This isn’t a symptom of “the web is too complex”; it’s a symptom of “the web is so widely used that it’s getting the security features which all software ought to have”.

                                                    The web is also too complex, but even if it was just basic HTML and CSS, we would want browsers sandboxed.

                                                    1. 2

                                                      Maybe the parent comment should be rewritten as, “The fact that we even need to worry about sandboxing for using untrusted input is embarrassing.” A lot of these problems would be solved if memory-safe languages were more widely used. (Probably not all of them, but a lot.)

                                                    2. 23

                                                      We need to worry about sandboxing for any file format that requires parsing, if it comes from an untrusted source and the parser is not written in a type-safe language. In the past, there have been web browser vulnerabilities that were inherited from libpng and libjpeg and were exploitable even on early versions of Mosaic that extended HTML 1.0 with the <img> tag. These libraries were written with performance as their overriding concern: when the user opens an image they want to see it as fast as possible and on a 386 even an optimised JPEG decoder took a user-noticeable amount of time to decompress the image. They were then fed with untrusted data and it turned out that a lot of the performance came from assuming well-formed files and broken with other data.

                                                      The reference implementation of MPEG (which everyone shipped in the early ’90s) installed a SIGSEGV handler and detected invalid data by just dereferencing things and hoping that it would get a segfault for invalid data. This worked very well for catching random corruption but it was incredibly dangerous in the presence of an attacker maliciously crafting a file.

                                                      1. 8

                                                        when the user opens an image they want to see it as fast as possible and on a 386 even an optimised JPEG decoder took a user-noticeable amount of time to decompress the image.

                                                        Flashback to when I was a student with a 386 with 2MB of RAM and no math co-processor. JPGs were painful until I found an application that used lookup-tables to speed things up.

                                                        1. 6

                                                          The reference implementation of MPEG (which everyone shipped in the early ’90s) installed a SIGSEGV handler and detected invalid data by just dereferencing things and hoping that it would get a segfault for invalid data. This worked very well for catching random corruption but it was incredibly dangerous in the presence of an attacker maliciously crafting a file.

                                                          I think this tops the time you told me a C++ compiler iteratively read linker errors.

                                                          1. 2

                                                            We need to worry about sandboxing for any file format that requires parsing, if it comes from an untrusted source and the parser is not written in a type-safe language.

                                                            But the vulnerabilities in the web are well beyond this pretty low-level issue.

                                                            1. 4

                                                              Bad form to reply twice, but I realise my first reply was just assertion. [Google Project Zero categorised critical vulnerabilities in Chrome and found that 70% of them were memory safety bugs]. This ‘pretty low-level issue’ is still the root cause of the majority of security vulnerabilities in shipping software.

                                                              1. 3

                                                                They often aren’t. Most of them are still memory safety violations.

                                                            2. 21

                                                              The web is an application platform now. It’s wasn’t planned that way, it’s suboptimal, but the fact that it is has to be addressed.

                                                              1. 17

                                                                Considering web browsers to be “glorified text documents” is reductive. We have long passed the point where this is the priority of the web, or the intention of most of it’s users. One cannot just ignore that a system that might have been designed for one thing 30 years ago, now has to consider the implications of how it has changed over time.

                                                                1. 5

                                                                  The fact that we even need to worry about sandboxing for looking at glorified text documents is embarrassing.

                                                                  Web browsers are now a more complete operating system than emacs.

                                                                  1. 4

                                                                    The fact that sandboxing is still a typical strategy, as opposed to basic capability discipline, is embarrassing. The fact that most programs are not capability-safe, let alone memory-safe, is embarrassing. The fact that most participants aren’t actively working to improve the state of the ecosystem, but instead produce single-purpose single-use code in exploitable environments, is embarrassing.

                                                                    To paraphrase cultural critic hbomberguy, these embarrassments are so serious that we refuse to look at them directly, because of the painful truths. So, instead, we reserve our scorn for cases like this one with Web browsers, where the embarrassment is externalized and scapegoats are identifiable. To be sarcastic: Is the problem in our systemic refusal to consider safety and security as fundamental pillars of software design? No, the problem is Google and Mozilla and their employees and practices and particulars! This is merely another way of lying to ourselves.

                                                                    1. 4

                                                                      It’s text documents from unknown/untrusted sources over a public network. If you knew exactly what the text document contained a priori you wouldn’t need to blindly fetch it over the internet, no?

                                                                      To me the issue is we’ve relaxed our default trust (behaviorally speaking) to include broader arbitrary code execution via the internet…but the need for trust was always there even if it’s “just text.”

                                                                    1. 10

                                                                      One of the common complaints about Lisp is that there are no libraries in the ecosystem. As you see, five libraries are used just in this example for such things as encoding, compression, getting Unix time, and socket connections.

                                                                      Wait are they really making an argument of “we used a library for getting the current time, and also for sockets” as if that’s a good thing?

                                                                      1. 16

                                                                        Lisp is older than network sockets. Maybe it intends to outlast them? ;)

                                                                        More seriously, Lisp is known for high-level abstraction and is perhaps even more general than what we usually call a general purpose language. I could see any concrete domain of data sources and effects as an optional addition.

                                                                        In the real world, physics constants are in the standard library. In mathematics, they’re a third party package.

                                                                        1. 12

                                                                          Lisp is older than network sockets.

                                                                          Older than time, too.

                                                                          1. 1

                                                                            Common Lisp is not older than network sockets, so the point is moot I think.

                                                                            1. 1

                                                                              I don’t think so. It seems to me that it was far from obvious in 1994 that Berkeley sockets would win to such an extent and not be replaced by some superior abstraction. Not to mention that the standard had been in the works for a decade at that point.

                                                                          2. 5

                                                                            Because when the next big thing comes out it’ll be implemented as just another library, and won’t result in ecosystem upheval. I’m looking at you, Python, Perl, and Ruby.

                                                                            1. 4

                                                                              Why should those things be in the stdlib?

                                                                              1. 4

                                                                                I think that there are reasons to not have a high-level library for manipulating time (since semantics of time are Complicated, and moving it out of stdlib and into a library means you can iterate faster). But I think sockets should be in the stdlib so all your code can have a common vocabulary.

                                                                                1. 5

                                                                                  reasons to not have a high-level library for manipulating time

                                                                                  I actually agree with this; it’s extraordinarily difficult to do this correctly. You only have to look to Java for an example where you have the built-in Date class (absolute pants-on-head disaster), the built-in Calendar which was meant to replace it but was still very bad, then the 3rd-party Joda library which was quite good but not perfect, followed by the built-in Instant in Java 8 which was designed by the author of Joda and fixed the final few quirks in it.

                                                                                  However, “a function to get the number of seconds elapsed since epoch” is not at all high-level and does not require decades of iteration to get right.

                                                                                  1. 7

                                                                                    Common Lisp has (some) date and time support in the standard library. It just doesn’t use Unix time, so if you need to interact with things that use the Unix convention, you either need to do the conversion back and forth, or just use a library which implements the Unix convention. Unix date and time format is not at all universal, and it had its own share of problems back when the last version of the Common Lisp standard was published (1994).

                                                                                    It’s sort of the same thing with sockets. Just like, say, C or C++, there’s no support for Berkeley sockets in the standard library. There is some history to how and why the scope of the Common Lisp standard is the way that it is (it’s worth noting that, like C or C++ and unlike Python or Go, the Common Lisp standard was really meant to support independent implementation by vendors, rather than to formalize a reference implementation) but, besides the fact that sockets were arguably out of scope, it’s only one of the many networking abstractions that platforms on which Common Lisp runs support(ed).

                                                                                    We could argue that in 2021 it’s probably safe to say that BSD sockets and Unix timestamps have won and they might as well get imported in the standard library. But whether that’s a good idea or not, the sockets and Unix time libraries that already exist are really good enough even without the “standard library” seal of approval – which, considering that the last version of the standard is basically older than Spice Girls, doesn’t mean much anyway. Plus who’s going to publish another version of the Common Lisp standard?

                                                                                    To defend the author’s wording: their remark is worth putting into its own context – Common Lisp had a pretty difficult transition from large commercial packages to free, open source implementations like SBCL. Large Lisp vendors gave you a full on CL environment that was sort of on-par with a hosted version of a Lisp machine’s environment. So you got not just the interpreter and a fancy IDE and whatever, you also got a GUI toolkit and various glue layer libraries (like, say, socket libraries :-P). FOSS versions didn’t come with all these goodies and it took a while for FOSS alternatives to come up. But that was like 20+ years ago.

                                                                                    1. 2

                                                                                      However, “a function to get the number of seconds elapsed since epoch” is not at all high-level and does not require decades of iteration to get right.

                                                                                      GET-UNIVERSAL-TIME is in the standard. It returns a universal time, which is the number of seconds since midnight, 1 January 1900.

                                                                                      1. 2

                                                                                        Any language could ignore an existing standard and introduce their own version with its own flaws and quirks, but only Common Lispers would go so far as to call the result “universal”.

                                                                                      2. 1

                                                                                        However, “a function to get the number of seconds elapsed since epoch” is not at all high-level and does not require decades of iteration to get right.

                                                                                        Actually, it doesn’t support leap seconds so in that case the value repeats.

                                                                                      3. 1

                                                                                        Yeah but getting the current unix time is not Complicated, it’s just a call to the OS that returns a number.

                                                                                        1. 6

                                                                                          What if you’re not running on Unix? Or indeed, on a system that has a concept of epoch? Note that the CL standard has its own epoch, unrelated (AFAIK) to OS epoch.

                                                                                          Bear in mind that Common Lisp as a standard, and a language, is designed to be portable by better standards than “any flavour of Unix” or “every version of Windows since XP” ;-)

                                                                                          1. 1

                                                                                            Sure, but it’s possible they were using that library elsewhere for good reasons.

                                                                                        2. 3

                                                                                          In general, I really appreciate having a single known-good library promoted to stdlib (the way golang does). Of course, there’s the danger that you standardise something broken (I am also a ruby dev, and quite a bit of the ruby stdlib was full of footguns until more-recent versions).

                                                                                          1. 1

                                                                                            Effectively that’s what happened though. The libraries for threading, sockets etc converged to de facto standards.

                                                                                      1. 24

                                                                                        I am very familiar with this smell from my $megacorp days and it smells of something brown and sticky and I don’t mean a stick.

                                                                                        First of all, this, and the whole letter, attempts to present a justification for why the experiment was carried out in this manner in terms of “this is the only way in which it could have been efficiently carried out”. Even if that were the case, and there really is no other way to study the efficiency of a review process with 100% accuracy (which is extremely doubtful IMHO), that is no justification for carrying it out in the first place. Some experiments simply cannot be carried out in an ethical manner. Tough luck, but there are always going to be other things to write papers about.

                                                                                        But the justification for not kickstarting any kind of discussion and not getting consent is pretty dubious:

                                                                                        we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches.

                                                                                        Deliberately sending buggy patches is not the only way to study the review process. You can do an “offline” study and look at regressions, for example, where the harm is already done. Or you can carry out the study “in vitro”, where consenting maintainers review different patch streams, some of which do include “hypocrite commits”. They will be on the lookout for bad patches but you will have a baseline based on the false positive figures from the maintainers who only got good patches. Either the false positive figures will be good enough to draw some conclusions about the efficiency of the review process, or they will indeed show that knowledge about the possible presence of bad patches influences the review process to the point where the experiment is useless. But in that case you at least have a plausible justification for a “blinder” study, and you can discuss better methods with the community.

                                                                                        IMHO, given the large volume of patches in some subsystems, and the fact that maintainership is hierarchical to some degree, I also suspect that it’s possible to co-opt the maintainers into this experiment, if you do it over a long enough timeframe. But suppose this really isn’t the case – this argument would’ve seemed plausible if other avenues had been investigated, but this apology letter doesn’t mention any such investigation.

                                                                                        It’s also not just about the ethics of this particular experiment. And also not just about the ethics, but the results, too.

                                                                                        Linux sees a lot of participation from so many different parties with so many conflicting interests (to cite just one example, many countries have various bans into place for Huawei’s telecom gear, but are otherwise using telecom gear that runs Linux which receives plenty of patches from Huawei, some of them quite, uh, controversial). So research into these processes is definitely relevant.

                                                                                        But without scrutiny and without good barriers in place, all you get is half-assed, under-the-radar attempts like this one, which are not only going to get a big backlash from the community, but are also not going to give very relevant results, because you can’t get the funding and set up a relevant, sufficiently large experiment while keeping it quiet. At least not in a public setting, where you have to publish or perish.

                                                                                        1. 6

                                                                                          An IRB that evaluates studies using human subjects will sometimes consider requests for use of deception or lack of informed consent. See, for example, this IRB’s guidelines on research involving deception. However, it’s quite difficult to get approval for use of deception or waiver of informed consent, and the researcher would have to make a very good case that either the harm to the subjects is minimal or that the research couldn’t be carried out otherwise. A good IRB would question whether alternative solutions, like the one you describe, could be feasible and still accomplish the goals of the research project. In the “hypocrite commits” study, as I understand it, they didn’t even seek IRB approval.

                                                                                          1. 2

                                                                                            I’m not too familiar with federal legislation (I’m, uh, across the poind) but IIRC consent is required even in research that involves deceptive methodology. You’re still required to get consent, and while it’s not required, it’s considered good practice to include a “some details to follow later” note on the consent form.

                                                                                            I am fairly sure (but see the note about being across the pond above…) that this study wouldn’t have gone past the IRB in its current form, because one of the key requirements for approval of deceptive methodology is proof that the study cannot be carried out without the waiver. It’s possible that the experiment could not have been carried out otherwise, but you’re required to show that. The paper doesn’t even discuss this aspect – it looks like they didn’t even bother to seriously consider it.

                                                                                            1. 1

                                                                                              A waiver or alteration of informed consent is possible in the US, you can see the rules at (f)(3) here. Among the requirements are that the research involves no more than minimal risk and that it can’t be carried out without the waiver.

                                                                                          2. 3

                                                                                            Even if that were the case, and there really is no other way to study the efficiency of a review process with 100% accuracy

                                                                                            Which isn’t even a requirement! Your research can be 95% accurate, and you can still get results from it and come up with new things to research based on that. Instead, they made it all-or-nothing. In which case, you’re right, the choice should have been nothing, not all.

                                                                                            1. 2

                                                                                              You can do an “offline” study and look at regressions, for example, where the harm is already done.

                                                                                              You can first lead a small observational study retrospectively like this but you’re going to need to run a controlled experiment to have more confidence that the results aren’t just statistical flukes and biases. Experimenting on humans without consent is obviously really bad. The arrogant and tone deaf response from the PI is just something else.

                                                                                              1. 3

                                                                                                You can first lead a small observational study retrospectively like this but you’re going to need to run a controlled experiment to have more confidence that the results aren’t just statistical flukes and biases.

                                                                                                Of course, but it’s a lot easier to get community support if you lead with “this is what we tried, and this is what we’d like to try in order to get better results” rather than “lol hello lab rat” and end with a letter of apology that can be summed up as sorry we got caught.

                                                                                                I don’t think the Linux kernel development community lacks awareness of just how finicky the review process is. It would’ve been very easy to get them on board with a study conducted in good faith. The fact that these guys didn’t even try isn’t even disappointing, it’s maddening.

                                                                                            1. 38

                                                                                              I’ve only been following the situation, but, while this is a reasonable act of contrition, I still want to see what UMN reports officially, because there’s clearly something up with their ethics policies if this slipped under the radar.

                                                                                              “We made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission”

                                                                                              That quote is exactly the root of it. You knew better, and you knew this wouldn’t fly. So you did it anyway. This seems pretty ‘reckless’, as per the legal term.

                                                                                              1. 19

                                                                                                Of course they could ask permission. They could have written a letter to Greg K-H: y

                                                                                                “Here are our credentials, here is what we propose to do, here is how to identify commits that should not be merged; we will keep track of them here and remind you at the end of each merge window; we will review progress with you on this data and this date; the project will end on this date.

                                                                                                Assuming you are generally agreeable, what have we overlooked and what changes should we make in this protocol?”

                                                                                                Doing that would make the professor and his grad students white-hats.

                                                                                                Not doing that makes them grey-hats at best, and in practice black hats because they didn’t actually clean up at the end.

                                                                                                People who have betrayed basic ethics: the professor, his grad students, the IRB that accepted that this wouldn’t affect humans, and the IEEE people who reviewed and accepted the first paper.

                                                                                                1. 9

                                                                                                  I’m not going to speak too much out of place, but academia does have an honesty and integrity problem in addition to an arrogance one. There is much debate these days within the context of the philosophy of science in how our metric fuelled obsession can be corrected.

                                                                                                  1. 7

                                                                                                    we knew we could not ask the maintainers of Linux for permission

                                                                                                    Their story continues to change. In the paper, they clearly state that they thought that their research didn’t involve human subjects and therefore they didn’t have to consent their participants. Then, it was that they asked for IRB review and the work was exempt from IRB guidelines, and now they are apologizing for non consenting people and that they knew that consenting people would have put their research at risk? Informed consent is one of the corner stones of modern (as in since the 60s) ethical research guidelines.

                                                                                                    There are ways around informed consent (and deceptive research practices, see here for an example) but those require strict IRB supervision and the research team needs to show that the benefits to participants and/or the community outweigh the risks. Plus, participants need to be informed about the research project after their participation and they still need to consent to participate in it.

                                                                                                    In before they didn’t know, all federal funding agencies (including NFS that is funding this work), requires all PIs and research personnel (e.g., research assistants) to complete training every 3 years on IRB guidelines and all students are required to complete a 12 hour course on human subject research (at least at my institution, I imagine that UMN has similar guidelines).

                                                                                                    1. 5

                                                                                                      But they are only students. It’s the people who have supervised them to blame. I hope for a positive outcome for all parts.

                                                                                                      1. 24

                                                                                                        PhD students are functioning adults akin to employees not k-12 students (juveniles), their supervisor and university deserves a share of the blame, so do they. Like if they had been employees their company and manager would deserve a share of the blame, and so would they.

                                                                                                        1. 16

                                                                                                          FWIW, the author of the email is the professor running the study and supervising the students who pushed patches out.

                                                                                                          I too am waiting to see if UMN does anything, either at the department level or higher.

                                                                                                          1. 3

                                                                                                            They applied for (and received) grants. They’re scholars, not students.

                                                                                                        1. 49

                                                                                                          As the saying goes:

                                                                                                          The first rule of distributing systems is don’t, if you can avoid it.

                                                                                                          What’s been amazing to me has been the amount of time and effort spent on this technique outside of the small fraction of places it makes sense, all while it is trivially obvious for anybody who stops and thinks for teo minutes what the problems are gonna be.

                                                                                                          The conference-industrial complex sold snake oil and everybody just lapped it up. Such is the quality of “engineering” in software.

                                                                                                          1. 29

                                                                                                            And worse, the snake oil often claims to cure a disease most people don’t have in the first place: the need to scale up dramatically on extremely short notice.

                                                                                                            1. 33

                                                                                                              I read a joke somewhere (I cannot remember where): “kubernetes. An ancient Greek word for more containers than customers.”

                                                                                                              1. 4

                                                                                                                I believe @srbaker coined that phrase.

                                                                                                                1. 3

                                                                                                                  Corey from Screaming in the Cloud (https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/) has a variant of this (if I’m remembering well):

                                                                                                                  Kubernetes, the Greek god of spending money in the cloud

                                                                                                                  1. 2

                                                                                                                    Boss wants to market horizontally scaling when vertical would be 10x cheaper :)

                                                                                                              1. 1

                                                                                                                I think the point about mathematical programming languages is missing something: is it haskell or mathematica?

                                                                                                                Take the example given for &&:

                                                                                                                True  && x     = x
                                                                                                                x     && True  = x
                                                                                                                False && False = False
                                                                                                                

                                                                                                                In the context of a haskell-like language, this is syntactic sugar for something like this:

                                                                                                                && x y = match (x,y) among
                                                                                                                         | (True,x) -> x
                                                                                                                         | (x,True) -> x
                                                                                                                         | (False,False) -> False
                                                                                                                

                                                                                                                On the other hand in a mathematica-like language, every one of those initial rules is a pattern which the evaluation engine may match, at its own discretion. It doesn’t desugar to anything.

                                                                                                                In a certain sense, this is an even stronger way to abstract over evaluation (in the manner discussed by TFA), but it is also accompanied by a loss of determinism.

                                                                                                                1. 1

                                                                                                                  To go in another direction, a C program is a precise mathematical formula whose meaning is expressed in terms of C abstract machine.

                                                                                                                  I don’t see a sharp distinction between math/non math in programming languages. Rather, I see different notational conveniences some of which are easier for humans to reason about.

                                                                                                                  1. 1

                                                                                                                    When i picture the engineering crew updating the LCARS widgets on star trek, i imagine them entering instructions in some mathematical/declarative language. It needs to be the ultimate language to transform idea to program by simply writing the idea down as closely as possible to how it goes in your head. This includes concise notation.

                                                                                                                    1. 9

                                                                                                                      It needs to be the ultimate language to transform idea to program by simply writing the idea down as closely as possible to how it goes in your head.

                                                                                                                      The trouble is, the idea in my head usually starts out being short-sighted and inconsistent until I spend days trying to squeeze it through the filter that’s a precise specification, which is basically any programming language.

                                                                                                                      1. 1

                                                                                                                        It does, but when editing code the existing formulation and language makes a huge difference to how quickly you can get there; some edits take minutes to make while others take weeks.

                                                                                                                        While this is primarily a function of the existing formulation, the availability of language features (eg exhaustiveness checking) makes a big difference too.

                                                                                                                      2. 6

                                                                                                                        That’s one vision of the future of software.

                                                                                                                        Another is the need for a “software archeologist” as in Vinge’s A Deepness in the Sky, who spelunks through the untold depths of accumulated cruft to find the routine needed.

                                                                                                                        I know which future I believe it most plausible.

                                                                                                                        1. 2

                                                                                                                          That part of Deepness is among my favorite predictions in any sci fi I’ve read.

                                                                                                                          1. 1

                                                                                                                            As someone who has just yesterday spent around an hour discussing with coauthors what exactly can be extracted from some paper pointed out by reviewers (a lot, but not what we need)… I see no reason to find these predictions in any kind of contradiction.

                                                                                                                            I expect some checks to demand you clarify your thoughts before them becoming executable, but I also expect some automatic checks to the mathematical languages used for preparing maths articles…

                                                                                                                      1. 4

                                                                                                                        Math is value oriented programming.

                                                                                                                        Oh and here we go again with the flawed analogies. So let’s dig deeper.. a deterministic view of the universe will be able to have a function describing the state of any particle/wave as a function of time. Another view is that things are scholastically mutating with time. Functions are not even primitive though if you consider them a special case of relations between sets.. and so on. Haskell does just fine with lambda calculus and category theory on the extreme end.

                                                                                                                        One of the reasons why a program written in functional programming is less complex than an object-oriented program is because, as we have just seen, it is more complex to define the sameness of objects than the sameness of values.

                                                                                                                        It’s less complicated because it tries its best to decouple code from state.

                                                                                                                        1. 1

                                                                                                                          When we decouple code from state, most of our code manipulates state.

                                                                                                                          1. 1

                                                                                                                            State is unavoidable but it’s best to contain it than spread it around like a toddler throwing spaghetti. Imagine writing a sql operation that saved state between calls? Yuck.. except that’s what ORM does in a sense :P

                                                                                                                        1. 50

                                                                                                                          The paper has this to say (page 9):

                                                                                                                          Regarding potential human research concerns. This experiment studies issues with the patching process instead of individual behaviors, and we do not collect any personal information. We send the emails to the Linux community and seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.

                                                                                                                          [..]

                                                                                                                          Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.

                                                                                                                          I’m not familiar with the generally accepted standards on these kind of things, but this sounds rather iffy to me. I’m very far removed from academia, but I’ve participated in a few studies over the years, which were always just questionaries or interviews, and even for those I had to sign a consent waiver. “It’s not human research because we don’t collect personal information” seems a bit strange.

                                                                                                                          Especially since the wording “we will have to report this, AGAIN, to your university” implies that this isn’t the first time this has happened, and that the kernel folks have explicitly objected to being subject to this research before this patch.

                                                                                                                          And trying to pass off these patches as being done in good faith with words like “slander” is an even worse look.

                                                                                                                          1. 78

                                                                                                                            They are experimenting on humans, involving these people in their research without notice or consent. As someone who is familiar with the generally accepted standards on these kinds of things, it’s pretty clear-cut abuse.

                                                                                                                            1. 18

                                                                                                                              I would agree. Consent is absolutely essential but just one of many ethical concerns when doing research. I’ve seen simple usability studies be rejected due to lesser issues.

                                                                                                                              It’s pretty clear this is abuse.. the kernel team and maintainers feel strongly enough to ban the whole institution.

                                                                                                                              1. 10

                                                                                                                                Yeah, agreed. My guess is they misrepresented the research to the IRB.

                                                                                                                                1. 3

                                                                                                                                  They are experimenting on humans

                                                                                                                                  This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                                                                                                                                  1. 19

                                                                                                                                    A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                                                                                                                                    1. 0

                                                                                                                                      A normal penetration testing gig is not academic research though. You need to separate between the two, and also hold one of them to a higher standard.

                                                                                                                                      This statement is so vague as to be almost meaningless. In what relevant ways is a professional penetration testing contract (or, more relevantly, the associated process) different from this particular research project? Which of the two should be held to a higher standard? Why? What does “held to a higher standard” even mean?

                                                                                                                                      Moreover, that claim doesn’t actually have anything to do with the comment I was replying to, which was claiming that this project was “experimenting on humans”. It doesn’t matter whether or not something is “research” or “industry” for the purposes of whether or not it’s “human experimentation” - either it is, or it isn’t.

                                                                                                                                      1. 18

                                                                                                                                        Resident pentester and ex-academia sysadmin checking in. I totally agree with @Foxboron and their statement is not vague nor meaningless. Generally in a penetration test I am following basic NIST 800-115 guidance for scoping and target selection and then supplement contractual expectations for my clients. I can absolutely tell you that the methodologies that are used by academia should be held to a higher standard in pretty much every regard I could possibly come up with. A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data.

                                                                                                                                        Let’s put it in real terms, I am hired to do a security assessment in a very fixed highly focused set of targets explicitly defined in contract by my client in an extremely fixed time line (often very short… like 2 weeks maximum and 5 day average). Guess what happens if social engineering is not in my contract? I don’t do it.

                                                                                                                                        1. 1

                                                                                                                                          Resident pentester and ex-academia sysadmin checking in.

                                                                                                                                          Note: this is worded like an appeal to authority, although you probably don’t mean it that way, so I’m not going to act like you are.

                                                                                                                                          I totally agree with @Foxboron and their statement is not vague nor meaningless.

                                                                                                                                          Those are two completely separate things, and neither is implied by the other.

                                                                                                                                          their statement is not vague nor meaningless.

                                                                                                                                          Not true - their statement contained none of the information you just provided, nor any other sort of concrete or actionable information - the statement “hold to a higher standard” is both vague and meaningless by itself…and it was by itself in that comment (or, obviously, there were other words - none of them relevant) - there was no other information.

                                                                                                                                          the methodologies that are used by academia should be held to a higher standard

                                                                                                                                          Now you’re mixing definitions of “higher standard” - GP and I were talking about human experimentation and ethics, while you seem to be discussing rigorousness and reproducibility of experiments (although it’s not clear, because “A penetration test does not create a custom methodology attempting do deal with outputting scientific and repeatable data” is slightly ambiguous).

                                                                                                                                          None of the above is relevant to the question of “was this a human experiment” and the closely-related one “is penetration testing a human experiment”. Evidence suggests “no” given that the term does not appear in that document, nor have I heard of any pentest being reviewed by an ethics review board, nor have I heard any mention of “human experimenting” in the security community (including when gray-hat and black-hat hackers and associated social engineering e.g. Kevin Mitnick are mentioned), nor are other similar, closer-to-human experimentation (e.g. A/B testing, which is far closer to actually experimenting on people) processes considered to be such - up until this specific case.

                                                                                                                                        2. 5

                                                                                                                                          if you’re an employee in an industry, you’re either informed of penetration testing activity, or you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment.

                                                                                                                                          if a company did this to their employees without any warning, they’d be shitty too, but the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific is nonzero.

                                                                                                                                          either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                                                                                                                          1. 1

                                                                                                                                            you’ve at the very least tacitly agreed to it along with many other things that exist in employee handbooks as a condition of your employment

                                                                                                                                            By this logic, you’ve also agreed to everything else in a massive, hundred-page long EULA that you click “I agree” on, as well as consent to be tracked by continuing to use a site that says that in a banner at the bottom, as well as consent to Google/companies using your data for whatever they want and/or selling it to whoever will buy.

                                                                                                                                            …and that’s ignoring whether or not companies that have pentesting done on them actually explicitly include that specific warning in your contract - “implicit” is not good enough, as then anyone can claim that, as a Linux kernel patch reviewer, you’re “implicitly agreeing that you may be exposed to the risk of social engineering for the purpose of getting bad code into the kernel”.

                                                                                                                                            the possibility that this kind of underhanded behavior in research could taint the results and render the whole exercise unscientific

                                                                                                                                            Like others, you’re mixing up the issue of whether the experiment was properly-designed with the issue of whether it was human experimentation. I’m not making any attempt to argue the former (because I know very little about how to do good science aside from “double-blind experiments yes, p-hacking no”), so I don’t know why you’re arguing against it in a reply to me.

                                                                                                                                            either way, the goals are different. research seeks to further the verifiability and credibility of information. industry seeks to maximize profit. their priorities are fundamentally different.

                                                                                                                                            I completely agree that the goals are different - but again, that’s irrelevant for determining whether or not something is “human experimentation”. Doesn’t matter what the motive is, experimenting on humans is experimenting on humans.

                                                                                                                                      2. 18

                                                                                                                                        This project claims to be targeted at the open-source review process, and seems to be as close to human experimentation as pentesting (which, when you do social engineering, also involves interacting with humans, often without their notice or consent) - which I’ve never heard anyone claim is “human experimentation”.

                                                                                                                                        I had a former colleague that once bragged about getting someone fired at his previous job during a pentesting exercise. He basically walked over to this frustrated employee at a bar, bribed him a ton of money and gave a job offer in return for plugging a usb key into the network. He then reported it to senior management and the employee was fired. While that is an effective demonstration of a vulnerability in their organization, what he did was unethical under many moral frameworks.

                                                                                                                                        1. 2

                                                                                                                                          First, the researchers didn’t engage in any behavior remotely like this.

                                                                                                                                          Second, while indeed an example of pentesting, most pentesting is not like this.

                                                                                                                                          Third, the fact that it was “unethical under many moral frameworks” is irrelevant to what I’m arguing, which is that the study was not “human experimentation”. You can steal money from someone, which is also “unethical under many moral frameworks”, and yet still not be doing “human experimentation”.

                                                                                                                                        2. 3

                                                                                                                                          If there is a pentest contract, then there is consent, because consent is one of the pillars of contract law.

                                                                                                                                          1. 1

                                                                                                                                            That’s not an argument that pentesting is human experimentation in the first place.

                                                                                                                                      3. 42

                                                                                                                                        The statement from the UMinn IRB is in line with what I heard from the IRB at the University of Chicago after they experimented on me, who said:

                                                                                                                                        I asked about their use of any interactions, or use of information about any individuals, and they indicated that they have not and do not use any of the data from such reporting exchanges other than tallying (just reports in aggregate of total right vs. number wrong for any answers received through the public reporting–they said that much of the time there is no response as it is a public reporting system with no expectation of response) as they are not interested in studying responses, they just want to see if their tool works and then also provide feedback that they hope is helpful to developers. We also discussed that they have some future studies planned to specifically study individuals themselves, rather than the factual workings of a tool, that have or will have formal review.

                                                                                                                                        They because claim they’re studying the tool, it’s OK to secretly experiment on random strangers without disclosure. Somehow I doubt they test new drugs by secretly dosing people and observing their reactions, but UChicago’s IRB was 100% OK with doing so to programmers. I don’t think these IRBs literally consider programmers sub-human, but it would be very inconvenient to accept that experimenting on strangers is inappropriate, so they only want to do so in places they’ve been forced to by historical abuse. I’d guess this will continue for years until some random person is very seriously harmed by being experimented on (loss of job/schooling, pushing someone unstable into self-harm, targeting someone famous outside of programming) and then over the next decade IRBs will start taking it seriously.

                                                                                                                                        One other approach that occurs to me is that the experimenters and IRBs claim they’re not experimenting on their subjects. That’s obviously bullshit because the point of the experiment is to see how the people respond to the treatment, but if we accept the lie it leaves an open question: what is the role played by the unwitting subject? Our responses are tallied, quoted, and otherwise incorporated into the results in the papers. I’m not especially familiar with academic publishing norms, but perhaps this makes us unacknowledged co-authors. So maybe another route to stopping experimentation like this would be things like claiming copyright over the papers, asking journals for the papers to be retracted until we’re credited, or asking the universities to open academic misconduct investigations over the theft of our work. I really don’t have the spare attention for this, but if other subjects wanted to start the ball rolling I’d be happy to sign on.

                                                                                                                                        1. 23

                                                                                                                                          I can kind of see where they’re coming from. If I want to research if car mechanics can reliably detect some fault, then sending a prepared car to 50 garages is probably okay, or at least a lot less iffy. This kind of (informal) research is actually fairly commonly by consumer advocacy groups and the like. The difference is that the car mechanics will get paid for their work where as the Linux devs and you didn’t.

                                                                                                                                          I’m gonna guess the IRBs probably aren’t too familiar with the dynamics here, although the researchers definitely were and should have known better.

                                                                                                                                          1. 18

                                                                                                                                            Here it’s more like keying someone’s car to see how quick it takes them to get an insurance claim.

                                                                                                                                            1. 4

                                                                                                                                              Am I misreading? I thought the MR was a patch designed to fix a potential problem, and the issue was

                                                                                                                                              1. pushcx thought it wasn’t a good fix (making it a waste of time)
                                                                                                                                              2. they didn’t disclose that it was an auto-generated PR.

                                                                                                                                              Those are legitimate complaints, c.f. https://blog.regehr.org/archives/2037, but from the analogies employed (drugs, dehumanization, car-keying), I have to double-check that I haven’t missed an aspect of the interaction that makes it worse than it seemed to me.

                                                                                                                                              1. 2

                                                                                                                                                We were talking about Linux devs/maintainers too, I commented on that part.

                                                                                                                                                1. 1

                                                                                                                                                  Gotcha. I missed that “here” was meant to refer to the Linux case, not the Lobsters case from the thread.

                                                                                                                                            2. 1

                                                                                                                                              Though there they are paying the mechanic.

                                                                                                                                            3. 18

                                                                                                                                              IRB is a regulatory board that is there to make sure that researchers follow the (Common Rule)[https://www.hhs.gov/ohrp/regulations-and-policy/regulations/common-rule/index.html].

                                                                                                                                              In general, any work that receives federal funding needs to comply with the federal guidelines for human subject research. All work involving human subjects (usually defined as research activities that involve interaction with humans) need to be reviewed and approved by the institution IRB. These approvals fall within a continuum, from a full IRB review (which involve the researcher going to a committee and explaining their work and usually includes continued annual reviews) to a declaration of the work being exempt from IRB supervision (usually this happens when the work meets one of the 7 exemptions listed in the federal guidelines). The whole process is a little bit more involved, see for example (all the charts)[https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts/index.html] to figure this out.

                                                                                                                                              These rules do not cover research that doesn’t involve humans, such as research on technology tools. I think that there is currently a grey area where a researcher can claim that they are studying a tool and not the people interacting with the tool. It’s a lame excuse that probably goes around the spirit of the regulations and is probably unethical from a research stand point. The data aggregation method or the data anonymization is usually a requirement for an exempt status and not a non-human research status.

                                                                                                                                              The response that you received from IRB is not surprising, as they probably shouldn’t have approved the study as non-human research but now they are just protecting the institution from further harm rather than protecting you as a human subject in the research (which, by the way, is not their goal at this point).

                                                                                                                                              One thing that sticks out to me about your experience is that you weren’t asked to give consent to participate in the research. That usually requires a full IRB review as informed consent is a requirement for (most) human subject research. Exempt research still needs informed consent unless it’s secondary data analysis of existing data (which your specific example doesn’t seem to be).

                                                                                                                                              One way to quickly fix it is to contact the grant officer that oversees the federal program that is funding the research. A nice email stating that you were coerced to participate in the research study by simply doing your work (i.e., review a patch submitted to a project that you lead) without being given the opportunity to provide prospective consent and without receiving compensation for your participation and that the research team/university is refusing to remove your data even after you contacted them because they claim that the research doesn’t involve human subjects can go a long way to force change and hit the researchers/university where they care the most.

                                                                                                                                              1. 7

                                                                                                                                                Thanks for explaining more of the context and norms, I appreciate the introduction. Do you know how to find the grant officer or funding program?

                                                                                                                                                1. 7

                                                                                                                                                  It depends on how “stalky” you want to be.

                                                                                                                                                  If NSF was the funder, they have a public search here: https://nsf.gov/awardsearch/

                                                                                                                                                  Most PIs also add a line about grants received to their CVs. You should be able to match the grant title to the research project.

                                                                                                                                                  If they have published a paper from that work, it should probably include an award number.

                                                                                                                                                  Once you have the award number, you can search the funder website for it and you should find a page with the funding information that includes the program officer/manager contact information.

                                                                                                                                                  1. 3

                                                                                                                                                    If they published a paper about it they likely included the grant ID number in the acknowledgements.

                                                                                                                                                    1. 1

                                                                                                                                                      You might have more luck reaching out to the sponsored programs office at their university, as opposed to first trying to contact an NSF program officer.

                                                                                                                                                  2. 4

                                                                                                                                                    How about something like a an Computer Science - External Review Board? Open source projects could sign up, and include a disclaimer that their project and community ban all research that hasn’t been approved. The approval process could be as simple as a GitHub issue the researcher has to open, and anyone in the community could review it.

                                                                                                                                                    It wouldn’t stop the really bad actors, but any IRB would have to explain why they allowed an experiment on subjects that explicitly refused consent.

                                                                                                                                                    [Edit] I felt sufficiently motivated, so I made a quick repo for the project . Suggestions welcome.

                                                                                                                                                    1. 7

                                                                                                                                                      I’m in favor of building our own review boards. It seems like an important step in our profession taking its reponsibility seriously.

                                                                                                                                                      The single most important thing I’d say is, be sure to get the scope of the review right. I’ve looked into this before and one of the more important limitations on IRBs is that they aren’t allowed to consider the societal consequences of the research succeeding. They’re only allowed to consider harm to experimental subjects. My best guess is that it’s like that because that’s where activists in the 20th-century peace movement ran out of steam, but it’s a wild guess.

                                                                                                                                                      1. 4

                                                                                                                                                        At least in security, there are a lot of different Hacker Codes of Ethics floating around, which pen testers are generally expected to adhere to… I don’t think any of them cover this specific scenario though.

                                                                                                                                                        1. 2

                                                                                                                                                          any so-called “hacker code of ethics” in use by any for-profit entity places protection of that entity first and foremost before any other ethical consideration (including human rights) and would likely not apply in a research scenario.

                                                                                                                                                    2. 23

                                                                                                                                                      They are bending the rules for non human research. One of the exceptions for non-human research is research on organization, which my IRB defines as “Information gathering about organizations, including information about operations, budgets, etc. from organizational spokespersons or data sources. Does not include identifiable private information about individual members, employees, or staff of the organization.” Within this exception, you can talk with people about how the organization merges patches but not how they personally do that (for example). All the questions need to be about the organization and not the individual as part of the organization.

                                                                                                                                                      On the other hand, research involving human subjects is defined as any research activity that involves an “individual who is or becomes a participant in research, either:

                                                                                                                                                      • As a recipient of a test article (drug, biologic, or device); or
                                                                                                                                                      • As a control.”

                                                                                                                                                      So, this is how I interpret what they did.

                                                                                                                                                      The researchers submitted an IRB approval saying that they just downloaded the kernel maintainer mailing lists and analyzed the review process. This doesn’t meet the requirements for IRB supervision because it’s either (1) secondary data analysis using publicly available data and (2) research on organizational practices of the OSS community after all identifiable information is removed.

                                                                                                                                                      Once they started emailing the list with bogus patches (as the maintainers allege), the research involved human subjects as these people received a test article (in the form of an email) and the researchers interacted with them during the review process. The maintainers processing the patch did not do so to provide information about their organization’s processes and did so in their own personal capacity (In other words, they didn’t ask them how does the OSS community processes this patch but asked them to process a patch themselves). The participants should have given consent to participate in the research and the risks of participating in it should have been disclosed, especially given the fact that missing a security bug and agreeing to merge it could be detrimental to someone’s reputation and future employability (that is, this would qualify for more than minimal risk for participants, requiring a full IRB review of the research design and process) with minimal benefits to them personally or to the organization as a whole (as it seems from the maintainers’ reaction to a new patch submission).

                                                                                                                                                      One way to design this experiment ethically would have been to email the maintainers and invite them to participate in a “lab based” patch review process where the research team would present them with “good” and “bad” patches and ask them whether they would have accepted them or not. This is after they were informed about the study and exercised their right to informed consent. I really don’t see how emailing random stuff out and see how people interact with it (with their full name attached to it and in full view of their peers and employers) can qualify as research with less than minimal risks and that doesn’t involve human subjects.

                                                                                                                                                      The other thing that rubs me the wrong way is that they sought (and supposedly received) retroactive IRB approval for this work. That wouldn’t fly with my IRB, as my IRB person would definitely rip me a new one for seeking retroactive IRB approval for work that is already done, data that was already collected, and a paper that is already written and submitted to a conference.

                                                                                                                                                      1. 6

                                                                                                                                                        You make excellent points.

                                                                                                                                                        1. IRB review has to happen before the study is started. For NIH, the grant application has to have the IRB approval - even before a single experiment is even funded to be done, let alone actually done.
                                                                                                                                                        2. I can see the value of doing a test “in the field” so as to get the natural state of the system. In a lab setting where the participants know they are being tested, various things will happen to skew results. The volunteer reviewers might be systematically different from the actual population of reviewers, the volunteers may be much more alert during the experiment and so on.

                                                                                                                                                        The issue with this study is that there was no serious thought given to what are the ethical ramifications of this are.

                                                                                                                                                        If the pen tested system has not asked to be pen tested then this is basically a criminal act. Otherwise all bank robbers could use the “I was just testing the security system” defense.

                                                                                                                                                        1. 8

                                                                                                                                                          The same requirement for prior IRB approval is necessary for NSF grants (which the authors seem to have received). By what they write in the paper and my interpretation of the circumstances, they self certified as conducting non-human research at time of submitting the grant and only asked their IRB for confirmation after they wrote the paper.

                                                                                                                                                          Totally agree with the importance of “field experiment” work and that, sometimes, it is not possible to get prospective consent to participate in the research activities. However, the guidelines are clear on what activities fall within research activities that are exempt from prior consent. The only one that I think is applicable to this case is exception 3(ii):

                                                                                                                                                          (ii) For the purpose of this provision, benign behavioral interventions are brief in duration, harmless, painless, not physically invasive, not likely to have a significant adverse lasting impact on the subjects, and the investigator has no reason to think the subjects will find the interventions offensive or embarrassing. Provided all such criteria are met, examples of such benign behavioral interventions would include having the subjects play an online game, having them solve puzzles under various noise conditions, or having them decide how to allocate a nominal amount of received cash between themselves and someone else.

                                                                                                                                                          These usually cover “simple” psychology experiments involving mini games or economics games involving money.

                                                                                                                                                          In the case of this kernel patching experiment, it is clear that this experiment doesn’t meet this requirement as participants have found this intervention offensive or embarrassing, to the point that they are banning the researchers’ institution from pushing patched to the kernel. Also, I am not sure if reviewing a patch is a “benign game” as this is the reviewers’ jobs, most likely. Plus, the patch review could have adverse lasting impact on the subject if they get asked to stop reviewing patches if they don’t catch the security risk (e.g., being deemed imcompetent).

                                                                                                                                                          Moreover, there is this follow up stipulation:

                                                                                                                                                          (iii) If the research involves deceiving the subjects regarding the nature or purposes of the research, this exemption is not applicable unless the subject authorizes the deception through a prospective agreement to participate in research in circumstances in which the subject is informed that he or she will be unaware of or misled regarding the nature or purposes of the research.

                                                                                                                                                          As their patch submission process was deceptive in nature, as their outline in the paper, exemption 3(ii) cannot apply to this work unless they notify maintainers that they will be participating in a deceptive research study about kernel patching.

                                                                                                                                                          That leaves the authors to either pursue full IRB review for their work (as a full IRB review can approve a deceptive research project if it deems it appropriate and the risk/benefit balance is in favor to the participants) or to self-certify as non-human subjects research and fix any problems later. They decided to go with the latter.

                                                                                                                                                      2. 35

                                                                                                                                                        We believe that an effective and immediate action would be to update the code of conduct of OSS, such as adding a term like “by submitting the patch, I agree to not intend to introduce bugs.”

                                                                                                                                                        I copied this from that paper. This is not research, anyone who writes a sentence like this with a straight face is a complete moron and is just mocking about. I hope all of this will be reported to their university.

                                                                                                                                                        1. 18

                                                                                                                                                          It’s not human research because we don’t collect personal information

                                                                                                                                                          I yelled bullshit so loud at this sentence that it woke up the neighbors’ dog.

                                                                                                                                                          1. 2

                                                                                                                                                            Yeah, that came from the “clarifiactions” which is garbage top to bottom. They should have apologized, accepted the consequences and left it at that. Here’s another thing they came up with in that PDF:

                                                                                                                                                            Suggestions to improving the patching process In the paper, we provide our suggestions to improve the patching process.

                                                                                                                                                            • OSS projects would be suggested to update the code of conduct, something like “By submitting the patch, I agree to not intend to introduce bugs”

                                                                                                                                                            i.e. people should say they won’t do exactly what we did.

                                                                                                                                                            They acted in bad faith, skirted IRB through incompetence (let’s assume incompetence and not malice) and then act surprised.

                                                                                                                                                          2. 14

                                                                                                                                                            Apparently they didn’t ask the IRB about the ethics of the research until the paper was already written: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf

                                                                                                                                                            Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. We apologize for the raised concerns. This is an important lesson we learned—Do not trust ourselves on determining human research; always refer to IRB whenever a study might be involving any human subjects in any form. We would like to thank the people who suggested us to talk to IRB after seeing the paper abstract.

                                                                                                                                                            1. 14

                                                                                                                                                              I don’t approve of researchers YOLOing IRB protocols, but I also want this research done. I’m sure many people here are cynical/realistic enough that the results of this study aren’t surprising. “Of course you can get malicious code in the kernel. What sweet summer child thought otherwise?” But the industry as a whole proceeds largely as if that’s not the case (or you could say that most actors have no ability to do anything about the problem). Heighten the contradictions!

                                                                                                                                                              There are some scary things in that thread. It sounds as if some of the malicious patches reached stable, which suggests that the author mostly failed by not being conservative enough in what they sent. Or for instance:

                                                                                                                                                              Right, my guess is that many maintainers failed in the trap when they saw respectful address @umn.edu together with commit message saying about “new static analyzer tool”.

                                                                                                                                                              1. 17

                                                                                                                                                                I agree, while this is totally unethical, it’s very important to know how good the review processes are. If one curious grad student at one university is trying it, you know every government intelligence department is trying it.

                                                                                                                                                                1. 8

                                                                                                                                                                  I entirely agree that we need research on this topic. There’s better ways of doing it though. If there aren’t better ways of doing it, then it’s the researcher’s job to invent them.

                                                                                                                                                                2. 7

                                                                                                                                                                  It sounds as if some of the malicious patches reached stable

                                                                                                                                                                  Some patches from this University reached stable, but it’s not clear to me that those patches also introduced (intentional) vulnerabilities; the paper explicitly mentions the steps that they’re taking steps to ensure those patches don’t reach stable (I omitted that part, but it’s just before the part I cited)

                                                                                                                                                                  All umn.edu are being reverted, but at this point it’s mostly a matter of “we don’t trust these patches and will need additional review” rather than “they introduced security vulnerabilities”. A number of patches already have replies from maintainers indicating they’re genuine and should not be reverted.

                                                                                                                                                                  1. 5

                                                                                                                                                                    Yes, whether actual security holes reached stable or not is not completely clear to me (or apparently to maintainers!). I got that impression from the thread, but it’s a little hard to say.

                                                                                                                                                                    Since the supposed mechanism for keeping them from reaching stable is conscious effort on the part of the researchers to mitigate them, I think the point may still stand.

                                                                                                                                                                    1. 1

                                                                                                                                                                      It’s also hard to figure out what the case is since there is no clear answer what the commits where, and where they are.

                                                                                                                                                                  2. 4

                                                                                                                                                                    The Linux review process is so slow that it’s really common for downstream folks to grab under-review patches and run with them. It’s therefore incredibly irresponsible to put patches that you know introduce security vulnerabilities into this form. Saying ‘oh, well, we were going to tell people before they were deployed’ is not an excuse and I’d expect it to be a pretty clear-cut violation of the Computer Misuse Act here and equivalent local laws elsewhere. That’s ignoring the fact that they were running experiments on people without their consent.

                                                                                                                                                                    I’m pretty appalled the Oakland accepted the paper for publication. I’ve seen paper rejected from there before because they didn’t have appropriate ethics review oversite.

                                                                                                                                                                1. 3

                                                                                                                                                                  Sometimes I need to make layer 7 protocols. I just run it over TLS and I force the use of v1.3 if possible. That is, I can rely on my language’s ssl module and its choices which must comply with TLSv1.3 anyway.

                                                                                                                                                                  Server-side, HAProxy can handle the TLS layer and load-balance the forwarded plain-text/octet protocol among local servers.

                                                                                                                                                                  The only thing is that you have to handle certificates. Fortunately, there is Let’s Encrypt and wildcards nowadays. I’m pretty sure there is still room for improvement, though. Maybe forcing only the strongest cipher suite allowed by TLSv1.3? Controlling session duration?

                                                                                                                                                                  1. 4

                                                                                                                                                                    Tls is solid and you probably can’t do better yourself. The place where nearly everyone fails is in how they manage their keys. You can have the strongest steel door in the world but if you leave the spare key under the potted plant then anyone can come in. Ditto for people generating weak and common keys due to poor entropy in both client and server.

                                                                                                                                                                  1. 9

                                                                                                                                                                    I do design crypto protocols for a living and I also do risk assessment and review for HSMs and various services. Getting it “right” is a matter of having a very strong engineering culture where security is not just left to a handful of wizards who stand up in defiance against product management. Everyone has to respect it as a foundational part of the development/manufacturing/deployment process.

                                                                                                                                                                    1. 3

                                                                                                                                                                      Gone are the days of dedicated and expensive hardware for many/most use cases. Commodity hardware is becoming sufficiently powerful with I/O and processing power. Next up, I’d like to see what Nginx can do.