Threads for lcapaldo

  1. 3

    Crash-tolerant software is indeed a virtue, but crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model. If every error is a terminal error then you kind of opt out of deterministic control flow.

    1. 1

      crash-only software often tends to encourage programming practices that make applications fiendishly difficult to model.

      Can you expand on that a bit?

      If every error is a terminal error then you kind of opt out of deterministic control flow.

      Well, there’s a bit more depth to it than that. For example, within the lifecycle of an individual request, experiencing a transient error (such as a timeout) might be fatal to that request, but not to the web server as a whole. Or for example, if your message queue consumer loses it’s connection; then you’d usually only restart the consumer, rather than the process as a whole.

      1. 1

        Is that what crash only means? All errors are crashes? My understanding was more that there was no happy/sad path when you terminated, a normal exit is indistinguishable from an abrupt one due to a crash, so any recovery happens in the startup path (and is constantly exercised).

        1. 6

          Going by Wikipedia definition:

          Crash-only software refers to computer programs that handle failures by simply restarting, without attempting any sophisticated recovery.

          The argument usually is: sophisticated (precise) error recovery is hard, and if you’re attempting it, you already have a potentially-broken program in an inconsistent state. Throwing it all away and starting from a blank state again is easier, well-tested, and therefore more robust.

          Take for example an out-of-memory error: if an allocation fails, you can either carefully backtrack the current operation and report the problem to the caller, or just abort() the whole program.

          I generally agree with the approach, but it’s not a silver bullet. Crash-only systems are prone to DoS-ing themselves. A persistent bad input can put the whole system in a crash loop, instead of being carefully skipped over. Even when everything technically works (and makes incremental progress as it should), the restart may be expensive/disruptive (e.g. loses caches or performs extra work on startup) and the system can still fail due to insufficient throughput.

          1. 1

            In a crash-only service, if an incoming request encounters a database invariant violation, does the entire process die? Or if the database connection fails, are all in-flight requests abandoned?

            Designing software so that it can start up and recover from a wide variety of prior states is a good idea in general, but it’s structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state. Any nontrivial DB transaction commit, for example, is a multi-stage operation. Same with network operations.

            More generally, it’s definitely a good idea to design the layer above the individual process to be resilient, but you can’t just assert that error handling isn’t a concern of the process. The process, the program, is what human beings need to mentally model and understand. That requires deterministic control flow.

            1. 2

              but you can’t just assert that error handling isn’t a concern of the process.

              I agree that is a bad idea, and I would almost say objectively so. Which is why I don’t think it is actually what the idea of “crash-only” is trying to convey.

              Again, my understanding was that crash-only software wasn’t “we crash on all errors/we don’t care about errors”, but rather “we don’t have shutdown code/the shutdown code isn’t where all the invariants are enforced”. It’s more about not having atexit handlers than not having catch blocks if you will. All program terminations are crashes, not all errors are crashes. If you have no shutdown code do you have to put that code somewhere else (periodic autosaves say, etc.), which means when you do crash you’re much likely to be closer to a recent good state.

              1. 1

                I may misunderstand what “crash-only” means. I take “crash” to mean “terminate the operating system process unceremoniously and without unwinding call stacks”, and I understand “only” not to mean all conceivable errors but definitely more than is strictly necessary.

              2. 1

                if an incoming request encounters a database invariant violation, does the entire process die? The original paper “Crash Only Software” talks about it in terms of individual components that can perform micro-reboots, and references Erlang heavily.

                So for an example, you’d want to use tools such as transactions so you can pretend that a multi-step operation is a single one. Alternatively, make everything idempotent, and retry a lot.

                structurally impossible to write a program that can crash at any point during its execution and reliably leave underlying resources in a recoverable state.

                I’m reasonably sure that this is what database software should be designed to do. Unless I’m somehow misunderstanding you.

                1. 1

                  There is no OS-level operation against e.g. disks, networks, etc. which can be interrupted at any arbitrary point, and can be reliably recovered-from. You can do your best to minimize the damage of that interruption – and maybe this is what crash-only is gunning for – but you can’t solve the problem. Every transition between layers of abstractions, between system models, represents an exchange of information over time. No matter how you design your protocol, no matter how tiny the “commit” or “sync” signal is, if your operation spans more than one layer of abstraction, and it’s possible for that operation to be interrupted, you can’t avoid the possibility of ending up in an invalid state. That’s fine! My point is only that systems at each layer of abstraction should not only do their best to recover from wonky initial state, but should also do their best to avoid making that wonky state in the first place. If you encounter an error, you should deal with it responsibly, to the best of your ability. That’s it.

          1. 1

            I use Vimwiki, especially the diary feature, I don’t bother to version control it (I have backups of course).

            1. 1

              Same. I don’t need to sync it to other machines and do regular scheduled borg backups including my vimwiki folder.

            1. 2

              Each server socket needs two file descriptors

              Is this always the case? I made an iocp/uring based http2 server, and it would be cool to claim it can handle 1 million concurrent connections.

              Also, even without uring, why isn’t one socket per connection enough? They allow for both reading and writing.

              1. 2

                More of the quote is:

                Each server socket needs two file descriptors:

                A buffer for sending

                A buffer for receiving

                You don’t need two file descriptors, (unless you want to count the listening fd which i don’t think they meant), seems like they were conflating the descriptor and the underlying socket, maybe by analogy with pipes, which do need 2 descriptors if you want bi-directional communication.

                1. 1

                  That’s my bad. I mis-remembered. For some reason i thought there are two file descriptors. After reviewing the docs, I realized I’m wrong. Both methods are only returning one file descriptor.

                  From beej’s networking guide:

                  accept() returns the newly connected socket descriptor, or -1 on error, with errno set appropriately.

                  The new socket descriptor to be used in subsequent calls, or -1 on error (and errno will be set accordingly).

                2. 2

                  There was a Slashdot article over 10 years ago about WhatsApp handling over a million connections on a single machine with FreeBSD / Erlang. Given that io_uring should be more efficient and computers are a lot faster, I would be pretty shocked if you couldn’t handle that many. I’d expect RAM for TLS protocol state to be your limiting factor, though if those connections are actually doing anything then network bandwidth might start to be (WhatsApp connections were mostly idle), though with 40 GigE that’s only 40 KB/s per connection and at that speed NUMA issues and the LLC sizes start to be important concerns (Netflix was saturating 40 GigE with TLS a few years back, the difficult thing was DMAing data into L3, encrypting it, and DMAing the encrypted data out again before the next DMA from disk started evicting things from L3 and slowed everything to a crawl).

                1. 3

                  Is this calling a private function or is it calling a public function with source identical to a private function? Is this a distinction without meaning in Erlang?

                  1. 2

                    It’s recompiling a module so that all functions are in fact “public” (i.e. exported), including previously “private” (un-exported) functions. Which isn’t the same as calling a private function from outside the module; by the time you call it here, it’s not private. Like @doug-moen says, it’s showing something about how you can dynamically recompile and hot-reload code in Erlang, which is rad for iterative development and live upgrades (though the upgrades get more complicated than I’ve ever dipped far into if you change data structures), but inevitably leads to this kind of thing. That’s all a part of the tooling, so I wouldn’t really see this as some kind of shady way of exposing supposedly private functionality but more a compromise that you have to make to get this kind of flexibility.

                    1. 1

                      It’s recompiling a module so that all functions are in fact “public” (i.e. exported), including previously “private” (un-exported) functions. Which isn’t the same as calling a private function from outside the module; by the time you call it here, it’s not private.

                      Right that’s what I am getting at. If there was a way to call a private function in Erlang that wasn’t “automate copy pasting the code”, would you be able tell the difference? I feel like the answer is no because there are no module scoped variables etc., whereas in python if you cloned a module to publicize a private function you could observe the difference between calling the copy and calling the original.

                      1. 1

                        There should be no difference.

                  1. 3

                    Thank you for “anti-unification” I needed to be able to name that.

                    1. 4

                      The idea of DNS lookup on Linux being implemented by libc is weird to me — shouldn’t something that fundamental be at a lower level?

                      That said, I believe Chrome has long had its own DNS resolver, probably so it can do async lookups.

                      As for Darwin — libc is part of libSystem, not a separate entity. And mDNSResolver is the daemon that runs multicast DNS and DNS-SD, aka Bonjour … but IIRC it might also act as the resolver/cache for regular DNS too.

                      Apple OSs have a framework called SystemConfiguration that manages settings, replacing a lot of text files in traditional Unix. It stores a lot of network config like DNS resolvers. One of the benefit is it has an observer API that lets you receive notifications of when settings change, which is very useful in a mobile system that switches networks a lot.

                      1. 4

                        shouldn’t something that fundamental be at a lower level?

                        What lower level? Are you expecting DNS related syscalls or maybe some /dev/dns device?

                        1. 3

                          Maybe a syscall, or at least a lower level library. Put it this way: since C standard libraries appear to be interchangeable components on Linux, I wouldn’t expect one of the tasks of building one to be implementing a DNS resolver. Just like I wouldn’t expect the stdio functions to implement a filesystem.

                          (For context, I’m coming from a world where the stdlib is part of the system library just like the syscall glue.)

                          1. 1

                            Just like I wouldn’t expect the stdio functions to implement a filesystem.

                            I don’t think a file system and a DNS resolver are at the same level of “fundamental”ness, you need the kernel to multiplex access to the storage for correctness and security’s sake with a file system, the socket related syscalls already provide sufficient multiplexing to implement DNS same as any other protocol that lives on top of tcp or udp.

                            Put it this way: since C standard libraries appear to be interchangeable components on Linux, I wouldn’t expect one of the tasks of building one to be implementing a DNS resolver.

                            From a “Linux” perspective it’s not that it is designed to have interchangeable C libraries, it’s that it is agnostic to whether there is a C library at all(*). Any pressures that result in multiple C libraries would apply equally to any “lower level” library IMO, they would be just as interchangeable.

                            And the lower level C library that implemented a DNS resolver would need syscall wrappers for socket etc. to do its work, wrappers which libc already has. There’s not much benefit to breaking it off.

                            You can imagine a “liblinux” that was just syscall wrappers and then a “libdns” that used that for just the resolver bits and then libc consumed both of those and provided stdio, et al on top of that, and that’s a fine design but there’s no way to force glibc and musl say to both use “liblinux”/“libdns” so that the “lower level” isn’t interchangeable but libc still is.

                            For context, I’m coming from a world where the stdlib is part of the system library just like the syscall glue

                            This is generally true in Linux, it’s just that the “system” library is called “libc”. glibc comes with a pile of syscall glue, musl comes with its own pile of syscall glue.

                            * This is of course a bit of hyperbole there are definitely things in Linux that are the way they are because they’ve been informed by the needs of implementing a posixy/unixy libc.

                        2. 3

                          The idea of DNS lookup on Linux being implemented by libc is weird to me — shouldn’t something that fundamental be at a lower level?

                          It’s turtles all the way down.

                        1. 1

                          Ok. Do we have any duplicates?

                          sort files.txt | uniq | wc

                          368435 ...

                          Not sure what the point of this is, unless you expect find to print the same path more than once for some reason?

                          Maybe you wanted something like <files.txt xargs -n 1 basename | sort | uniq | wc?

                          1. 1

                            Just to sanity-check, really. “Does this actually look the way I think it does? Ok good.”

                            1. 1

                              Fair enough.

                          1. 7

                            I agree this isn’t “programming”, but I used the suggest mechanism to suggest tagging it as a11y. The author is using this technology to help control how they present themselves, and it enables them to do that while including hand gestures, etc., which “just turn the camera off” doesn’t allow.

                            1. 4

                              NCommander created a 30 minute YouTube video looking into the truthfulness of this blog post, and while Raymond Chen isn’t lying, there might be more to the story than just the bugs he found…

                              https://youtu.be/3EPTfOTC4Jw

                              1. 7

                                There was a very recent follow up which covers this https://devblogs.microsoft.com/oldnewthing/20220106-00/?p=106122 and expect was the impetus for this post, the follow up was already submitted here: https://lobste.rs/s/tutknz/filling_some_gaps_story_space_cadet

                                1. 1

                                  Can you give an executive summary of that 30 minute video?

                                  1. 10

                                    Space Cadet is present on the installation disks for every single version of 64-bit Windows XP, but is manually disabled and is not automatically installed (although, curiously, it is installed in very late 64-bit XP builds). After extracting the game files and running the game, there are a couple minor graphical glitches but the game works entirely fine, despite what Raymond Chen claimed. He also reverse engineers the source of the floating point rounding error Raymond is talking about and speculates that the bug was fixed for one of the 64-bit versions without the dev teams knowing that this fix also fixed the bug on the other 64-bit architectures. Raymond later posted an updated second post where he responds and tells a story about how when the 64-bit porting process was happening, they only had access to long-discontinued 64-bit Alpha AXP processors with architecture-specific bugs which created far more severe collision issues than the ones NCommander managed to reverse engineer. When the Alpha AXP 64-bit dev port was then ported to Itanium and AMD64 these bugs went away, and Pinball was left with the more minor collision issues NCommander found, which both NCommander and Raymond speculate were caused by some compiler or C runtime floating-point precision bug, which one or the other of those teams later fixed, accounting for why Pinball existed on the later systems. Whoever added it back just never told Raymond that they got it working.

                                    1. 2

                                      Hah! Thanks.

                                1. 6

                                  Here to pick some nits:

                                  Startup Times Ulrich Drepper also said:

                                  With prelinking startup times for dynamically linked code is as good as that of statically linked code. Let’s see how true that is.

                                  Emphasis mine. Did you actually use prelinking? Apologies if so and you just weren’t explicit about it. I’ve seen some impressive improvements with it, though in those cases I did not have the option of static vs. dynamic vs. prelink to really compare head to head, just prelinked vs non-prelinked.

                                  [re ltrace] The truth is that you could do this with a proper build of a static library that tracked calls for you. In fact, that would give you more control because you could pick and choose which functions to trace at a fine-grained level.

                                  For the record ltrace lets you pick and choose which functions to trace at a fine grained level.

                                  1. 2

                                    Emphasis mine. Did you actually use prelinking?

                                    No, I did not. I was under the impression that glibc’s linker automatically did it. I will do it, update the post, and let you know.

                                    For the record ltrace lets you pick and choose which functions to trace at a fine grained level.

                                    Good to know. I will update the post.

                                    1. 2

                                      To update: I was not able to get prelinking to work, so I put a note in instead.

                                      1. 2

                                        I believe prelinking is now disabled because ASLR requires libraries to be linked at different addresses. I think OpenBSD has a prelinking-like thing that works with their form of address-space randomisation.

                                        Note that static linking means that you are completely opting out of this kind of randomisation. Even if you build a statically linked PIE (mmm, PIE) then the displacements from any function pointer to any gadget are the same. I remain unconvinced that ASLR is anything other than security theatre but it is something you give up with most static linking designs (there are some half-way places where you compile with -ffunction-sections and -fdata-sections and randomise the binary layout in between each launch).

                                  1. 4

                                    Whether using the define or the union approach I’m having a hard time imagining this being worth it. The extra .sub.field just doesn’t seem that painful if you need the bulk sub object handling as well. Maybe in a “it’s been 10 years of accretion this should have been a sub object all along, but I don’t want to touch all the consumers” kind of scenario? Does somebody have a concrete “in the wild” example?

                                    1. 1

                                      I found this interesting (along with a lot of the related posts and others on the site) but it felt like it was being presented as an alternative to something, but I had a hard time figuring out what that something was. DSLs seems vague. I think I finally got the idea from noticing the punny file name “caternetes.html”. The DSLs being referred to are at least in part things like k8s configurations or terraform etc.? Is that the idea? Analogously to the the post about writing code not configuration?

                                      1. 2

                                        Yes, that’s right. I kind of wanted to avoid being explicit about that, because I’m worried that if I call out any specific technology, fans of it will immediately dismiss this idea. :)

                                        But, your comment made me finally add a little (partial) list to the conclusion: http://catern.com/caternetes.html#conclusion where hopefully it will be less threatening.

                                      1. 3

                                        bsandro from IRC mentions this clump of usenet posts about long long.

                                        1. 4

                                          A lot of neat stuff in the thread.

                                          (But what about 128-bits: I’d be pleased to have a 128-bit type as well … however, a pragmatic view says: we have the 64-bit problem right now, we’ve had it for several years; we won’t have the 128-bit problem for quite a few years. Based on the typical 2 bits every 3 years increase in addressing, a gross estimate is: 32 bits/2 bits * 3 years = 48 years, or 1992+48 = 2040. Personally, I’m aggressive, so might say year 2020 for wanting 128-bit computers …

                                          I’m excited for the imminent arrival of long long long.

                                          1. 9

                                            One of my favourite error messages:

                                            $ cat t.c
                                            long long long l;
                                            $ cc -c t.c
                                            t.c:1:11: error: ‘long long long’ is too long for GCC
                                            
                                          2. 3

                                            As someone who was alive and working (at Sun) through much of this time the only thing I would add (that’s relevant to the OP) is that Microsoft was not known to engage with the community of the time (you might recall that their first internet ready OS was called Windows 95).

                                          1. 3

                                            Okay, I can see the emoji thing going for the same approach as diacritics where we can have say ü written both as the ü glyph (U+00FC) or an u glyph (U+0075) followed by an umlaut ¨ (U+00A8).

                                            So, are we having emoji normalization forms too? Because I really don’t want to be the guy tracking sorting and filtering bugs that failed to match bear + snowflake sequences when searching for polar bears.

                                            1. 5

                                              It’s not really the same thing. Normalization happens because both the composed and decomposed versions result in, logically, the same string. There is no ‘composed’ equivalent for these sequences; the only way to represent a polar bear is to put multiple codepoints together.

                                              1. 1

                                                No, but imagine you have a commutative case. E.G., what if snowflake+bear = bear+snowflake = polar bear? Then, what if a webpage contains snowflake+bear, but you search for bear+snowflake? You have to match the two somehow.

                                                1. 6

                                                  Good news: modifiers and JWZ sequences aren’t commutative. [Snowflake, JWZ, bear] just looks like ❄️🐻.

                                                  1. 3

                                                    You mean ZWJ right?

                                                    1. 6

                                                      No, no, Jamie Zawinski has been really busy lately.

                                                      1. 2

                                                        Seriously. There were plenty of search results for “jwz unicode” that the “did you mean” thing did not kick in. it took me a bit to figure out the initialism was probably accidentally transposed.

                                                        1. 1

                                                          You’ve correctly guessed why I transposed it too.

                                                        2. 2

                                                          I, for one, am in favor of a JWZ unicode character. It would probably be disappointment/rage with technology.

                                                        3. 2

                                                          Yes

                                                  2. 1

                                                    The fun thing is that “are these two sequences the same emoji” becomes undecidable then :D

                                                  1. 110

                                                    Independently of what burntsushi decides, I think that the following concrete actions would be in order:

                                                    1. Remove the last half-sentence of the banner that explains how to delete your account. Having it as the last part of the message adds a lot of weight ot it: after you read the banner it’s the main thing you remember, the oddly detailed explanation of how to erase yourself.

                                                    2. Reformulate the rest of the banner to indicate that this is a heuristic that may be wrong. It’s important when providing hints from data to tell people explicitly that we know the hints may be wrong. Otherwise it feels very different.

                                                    Let’s rephrase this banner with the explicit goal of doing no harm (better, doing good) when it is wrongly displayed, instead of thinking about the formulation when its assessment is correct.

                                                    1. 59

                                                      Speaking as somebody who’s gotten that banner basically permanently on their account, I think a lot of folks here may be missing important context:

                                                      • The call to delete your account makes a lot of sense if you have to handle users dramatically declaring, in DMs/IRC/elsewhere “One more and [clutches pearls] I shall leave, forthwith!”. This is tiresome from a moderation standpoint and annoying for a community, and if one can nudge somebody into removing their account–in the common case!–rather than continue to engage in a community they don’t fit into (for whatever reason), it’s a win for both parties.
                                                      • This isn’t some randy sketchy heuristic–the warning links directly to a histogram that shows where you are in the flaggings. It’s deterministic, it’s impartial, and if you’re getting flagged a lot there’s something going on with either your posting or the community.
                                                      • Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.
                                                      • One of the biggest reasons for Lobsters coming from the orange site was @jcs being annoyed at opaque moderation. While I might disagree certain particular actions, there is a public log. The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.
                                                      • There’s a trend for folks to go on IRC, Twitter, or wherever and talk trash about our community, about how @pushcx is powertripping, or whatever. This behavior probably plays well with the circles they’re familiar with and probably feels like punching up, but it’s gotta suck for the person who manages the community to have people he’s tried to work with and accomodate throw it back in his face.
                                                      • People are really flag-happy, and this is what happens when you are.

                                                      (Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

                                                      1. 35

                                                        Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.

                                                        This bit deserves signal boosting IMO.

                                                        Lobsters doesn’t cost anything to join or participate in. The people who run it are clearly doing it for love, not money.

                                                        Speaking for myself I’d much rather get past the momentary ouch of having a red warning message attached to my account than have the owners or moderators rage quit as a result of feeling UTTERLY drained by the experience.

                                                        I’m watching the life get sucked out of some very stalwart well meaning people in another community I care about simply due to the sheer VOLUME of constant negative feedback, so I think we all owe it to the mods to suck it up and cut him/them some serious amounts of slack.

                                                        1. 22

                                                          <3 Thank you very much.

                                                        2. 24

                                                          (Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

                                                          I would love to see this come back in some form. As someone who joined after that was removed, I feel there isn’t enough information about flags. Every single time I’ve ever flagged something, I’ve been really hesitant because I wasn’t able to find any guidance on what qualifies to be flagged.

                                                          1. 3

                                                            I agree. I actually don’t think this is so minor on its own though I can see why you phrased it that way in its original context.

                                                          2. 23

                                                            I realize I’m not as active as I might be - I’m bipolar, my ability to spend time on things comes and goes - but I promise that I do still pay active attention to the site and step in when it seems warranted. Also, the mod team has lots of discussions behind the scenes. Just because @pushcx is the face of a decision doesn’t mean the rest of us aren’t involved.

                                                            I felt I should address that, since you mentioned it.

                                                            Edit to add: Thank you for the kind words, overall. It’s helpful to know that you and others do see the point of the banner.

                                                            1. 22

                                                              People are really flag-happy, and this is what happens when you are.

                                                              Speaking from experience as someone who’s moderated some decent-sized subreddits – where reddit has both the ability to downvote, and a separate ability to “report” for posts/comments that break subreddit or site-wide rules – this is absolutely the case. The number of people who just spam-click the “report” button thinking it’s some sort of super-downvote that will remove the thing they dislike is astounding. Same with people who periodically decide they just don’t like some type of content even if the subreddit generally allows it, and go mass-report a hundred posts to express their displeasure.

                                                              1. 11

                                                                Yeah, this is what I think is fundamentally the issue. A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button. Perhaps in their mind, if the post was deleted, then the opinion no longer exists or is somehow invalid???

                                                                I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

                                                                People simply think people who disagree with them are either stupid, or trolling (deliberately saying stupid things hoping for a reaction) or both.

                                                                1. 12

                                                                  A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button.

                                                                  It’s also true that people sometimes repeatedly post tired nonsense, and that merely (and at obviously low cost) repeating something that’s poorly researched, or broadly offensive, doesn’t entitle the poster to a vigorous debate. Sometimes the use of a flag is the only viable option.

                                                                  1. 4

                                                                    If somebody posted some tired nonsense then it should be very easy to dispute.

                                                                    doesn’t entitle the poster to a vigorous debate

                                                                    You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

                                                                    Sometimes the use of a flag is the only viable option.

                                                                    Anybody could deem all your posts ‘tired nonsense’ and just report all of them? That is as legitimate as when you do it. Do you somehow think that your idea of what is tired nonsense is universal and therefore a good metric for when a post should be reported?

                                                                    1. 8

                                                                      If somebody posted some tired nonsense then it should be very easy to dispute.

                                                                      In the case of trolling, the whole point is to trick people into taking the bait, which I believe moves the subthread higher up in the scoring ranks.

                                                                      You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

                                                                      I think that detecting trolling is a skill that’s pretty key to the survival of a community, but a good troll will make it quite difficult to resist engaging. Trolling isn’t commonly overt, over the top rudeness. Way more often it is an attempt at looping other community members into a “debate” that ends up in people pointing edgy and charged hot takes at each other.

                                                                      In its worst form, it will promote socially harmful conclusions through pseudo-from-first-principles lines of reasoning that are aesthetically attractive to the type of people that visit tech forums. To dispute requires dissecting phony arguments, which at a glance, appears to legitimize the “debate” and grant the troll a certain level of community approval, especially if they’re good at giving off the appearance of rationalism. IMO ignoring and flagging this type of content is nowhere near what’s required fully address it.

                                                                      1. 3

                                                                        How are you not using an accusation of trolling as a free ‘You lose’?

                                                                        How do you know that something is true? It is via putting your ideas up for others to engage and debate. But you want to simply circumvent that by calling opinions you don’t like trolls, and then saying that those opinions don’t even need to be debated or engaged because that will legitimise it.

                                                                        Sooner or later instead of having attacking each other’s ideas so we can improve then, all we will end up doing is claiming the other side is ‘trolling’ so they are immediately wrong and don’t even need to be disproven.

                                                                        Why even bother defending your ideology against opposition, when you can simply claim the opposition is a foreign spy or a mentally ill person and then getting rid of them?

                                                                  2. 5

                                                                    I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

                                                                    Or maybe you’ve gotten more and more trollish over time.

                                                                    1. 5

                                                                      Or maybe we have too many politically driven ideologues who are not interested in communication and they simply throw the word troll around to avoid having to engage with their opposition on the idea plane.

                                                                      1. 9

                                                                        Nah, I’ve been around just about as long as you have, I can count on one hand the number of times I’ve not seen you be a troll in a thread. I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread. Pretty much all you do is get into fights around here.

                                                                        1. 3

                                                                          Devil’s Advocate: Perhaps @LibertarianLlama is among a new breed of Lobsters who are also concerned with the human side of things, not just the technical one.

                                                                          1. 2

                                                                            I’ve not seen you be a troll in a thread

                                                                            I suppose you are one of those people who have a very low margin for what constitutes ‘trolling’.

                                                                            I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread.

                                                                            I didn’t realise technical expertise in a subject is a requirement of using this website. Regardless you can just assume that I have no expertise in any subject.

                                                                    2. 6

                                                                      I wonder if a literal “super downvote” button would work as a psychological trick. It would act as a normal downvote, but just look like a bigger, angrier button that would absorb more anger from whoever clicks it. (At the same time, possibly rate-limit the actual report button per user…)

                                                                      1. 8

                                                                        That’s a pretty cute idea. I’m tempted to say that, for people who have the self awareness to realize that what they mostly need is catharsis, I recommend getting it from their favorite action videogame, or a trip to the gym or whatever… I don’t want to dismiss your idea though, it’s true that it could help to have some sort of reminder or prompt to seek catharsis.

                                                                        1. 3

                                                                          Edit: someone has looked at the system and much of this is actually already automated(!): https://lobste.rs/s/zp4ofg/lobster_burntsushi_has_left_site#c_qiipbc

                                                                          Previous:


                                                                          OK, another cute - or hopefully even better - useful idea:

                                                                          I think HN has a pr account setting that ignores flag from that user, I’m fairly certain dang has mentioned at some time that one should be careful with flags. (Now that I think about it it might have been the vouch option he was writing about.)

                                                                          I’m not really sure how it would work with perfectly transparent moderation:

                                                                          • would the user get a warning?
                                                                          • or would it just show up in the moderation logs?
                                                                          • maybe it is a fraction so if a user has 1/5 flag weight it takes more than 5 such users to flag before it counts as one ordinary user? Then the moderation log could still say “flagged based on user feedback” or whatever it say but omit the details about there being 10 flag abusers counting as two full votes and 3 ordinary users?
                                                                          1. 11

                                                                            Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

                                                                            Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

                                                                            1. 1

                                                                              Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

                                                                              Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

                                                                        2. 5

                                                                          The moderators of r/TheMotte (a subreddit which has the best moderation IMO) observed pretty much the same:

                                                                          No one who has seen a mod queue would be surprised by this. Even those of you who are always very calm and nonconfrontational, with middle-of-the-road opinions, might be shocked how often your posts are reported. Some of the most reasonable, unspicy posts get marked as “It’s targeted harassment.” Some people will report “Inflammatory claim without evidence” just because they don’t agree with the post.

                                                                          I pretty much started brushing aside (like several other users reportedly do) the red warning that lobste.rs throws in your profile for this reason, as well as for that condescending final phrase “Reconsider your behaviour or take a break” which rests on the erroneous assumption of there incontrovertibly being sound rationality in this community’s flagging behaviour.

                                                                          I have a theory that this became worse after the site removed the downvote functionality thus removing a potential anger-outlet for users, who have now shifted to channel that affective energy via flagging. This theory however can only confirmed by doing a simple analysis of the data (was there an increase in flagging right after disabling downvoting?)

                                                                        3. 12

                                                                          It’s irrelevant that it is impartial if it is exploitable. Which it very much is.

                                                                          So the system allows for a mob to harass an user into leaving the site, but you claim that it is important to note that anyone can be harassed. That is the matter subject to challenge here, not “important context”.

                                                                          Yes people are trigger happy, for that reason, the flagging feature might be counter productive.

                                                                          1. 5

                                                                            you claim that it is important to note that anyone can be harassed.

                                                                            I’m not sure I see where you got that from my post.

                                                                            The ability of the mob to harass users here exists, and is exploited, at scale and quite apart from this notice that “yo, you’re getting a significant number of flags.”

                                                                            It doesn’t make sense to optimize for people who run away when told others disagree with them.

                                                                            1. 21

                                                                              No one wants to be told they don’t belong here. I really enjoy Burntsushi and a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

                                                                              Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

                                                                              1. 6

                                                                                OK I see an important distinction here. I tried to point this out to burntsushi on Twitter.

                                                                                If you read the message he got VERY carefully, it sets forth a very specific sequence of potential and suggested routes to mitigation. They are:

                                                                                1. Take a breath. Take a break. Step away for a bit and think about whether maybe the problem might exist at least partly in your head and the resultant behavior you’re exhibiting in the community.

                                                                                2. Talk to a moderator about the situation.

                                                                                3. Then, and ONLY then, does the message point out that you can if you so choose also delete your account.

                                                                                Having gotten the big red warning myself a ways back, I DO sympathize that it’s an ouchy thing to have happen in the moment, but I strenuously believe that the intent here isn’t to make anyone feel unwelcome. Just the opposite, my read of the intent is an earnest attempt on the part of the community to help people moderate their behavior, and if that’s not possible or applicable, to seek other options for mitigation.

                                                                                1. 11

                                                                                  No one wants to be told they don’t belong here.

                                                                                  I’d agree, but we have done very poorly by some of our users in that regard already and the mob likes it that way. Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff? Peppridge farm remembers.

                                                                                  There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe–and they’ll get even angrier if you suggest that maybe, possibly, we should focus on something else.

                                                                                  So, while I agree in principle with you, the community has through its actions demonstrated that it does not care.

                                                                                  Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

                                                                                  I don’t believe that it is. You are, of course, welcome to your own interpretation.

                                                                                  1. 14

                                                                                    Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff?

                                                                                    IIRC he had a pattern of problematic behavior, and also I’m not sure how much of an expert in ESE he actually is: his book is overwhelming, but keeps giving me bad vibes. I’m struggling to explain what it actually is, but I’m really suspicious that there are serious problems with it.

                                                                                    1. 1

                                                                                      I’d definitely be interested to hear more.

                                                                                      I’m struggling to explain what it actually is

                                                                                      Mind giving it a try? I couldn’t see anything particularly bad (relatively speaking) in their comment history, nor anything related to the ban rationale. Seems like you’ve been paying more attention, haha

                                                                                    2. 14

                                                                                      There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe

                                                                                      The basic problem with statements like this is that nearly everyone can feel it applies to the angry mob that’s disagreed with them.

                                                                                      Personally I’m on record as believing that there is no way to separate “politics” from what we do (simple example 1, simple example 2) or to write software without considering the impact it will have on people.

                                                                                      But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue. The hypothetical guy (example 2 above) with no criminal record who can’t get bail because the system could only store his name as a bunch of “aliases” is “politics” to me, but apparently not to many other people (it’s just “falsehoods programmers believe about names”), while holding a software project’s leader accountable for misbehavior is “politics” to many of those people and common sense to me. A staggering percentage of actual “political” fights really are that sort of thing.

                                                                                      1. 3

                                                                                        But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue.

                                                                                        I mean the proof is at hand right? A “let’s not discuss politics” non sequitur as successful troll.

                                                                                        1. 0

                                                                                          But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted

                                                                                          I don’t think I’ve ever seen this on Lobsters, only on others sites. Do you have any examples?

                                                                                        2. 1

                                                                                          I think we disagree about what it means to be thin skinned. I thought you were admonishing the people who leave into toughening up. If that is the case, I think it will lead to either an echo chamber or a dungeon of trolls or both.

                                                                                          But based on your other comments, I don’t think you meant what I interpreted.

                                                                                          You in this context means, us, or the offended user that flags people into quitting. How about we make flags public and you can only spend them in certain ways? Do comment points even mean anything? There is a huge distinction between “this person is harmful” and basically anything else. if someone is harmful, they should be removed. Otherwise … if you don’t want to see their posts, then that would be a personal choice.

                                                                                          1. 2

                                                                                            Making flags public is a terrible idea.

                                                                                            1. 2

                                                                                              I agree. I DO think that people here are overly trigger happy with the flagging, but given other comments in this thread I’m willing to believe this is simply human nature at plan and unavoidable. Making flags public would be in effect shaming someone for trying to make a small contribution to the community by flagging a post.

                                                                                              1. 2

                                                                                                I feel that maybe 60% of flags are not trying to contribute to the community, but rather to avenge themselves, or to have an outlet for disagreement without having to engage. It’s part of the problem with the Internet at large: for digital natives, disengagement is no longer an option. People can’t just ignore stuff they disagree with; they have to respond to it, and if it’s a troll, they shouldn’t engage directly, so they instead engage indirectly by flagging. There’s a reason why Twitter’s first option when reporting is “I disagree with this”. It’s just human nature.

                                                                                              2. 1

                                                                                                Because? Without justification your statement is opinion, what am I to do with that?

                                                                                                1. 1

                                                                                                  Because it can lead to retaliatory, tit-for-tat flagging of the flagger’s comments. Mods can already see who flags what and can take action against those who misuse it.

                                                                                          2. 2

                                                                                            a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

                                                                                            Who are “people like that”? To me, this sentence seems to extrapolate based on a single data point, which is based on some observations of another person. That is to say, we don’t know Burntsushi well enough, and maybe no matter how we setup the system, he would find something to quit. And striving to optimize for one would put burden and alienate others.

                                                                                            1. 5

                                                                                              people like that

                                                                                              Hardworking, smart, helpful, writes a lot of great Rust. Reddit and github

                                                                                              I am talking about specifics not a generalized idea of a Lobste.rs user.

                                                                                            2. 0

                                                                                              Since that person left, I’m not getting a flag on every single of my comments anymore.

                                                                                              An interesting coincidence. ¯\_(ツ)_/¯

                                                                                              1. 0

                                                                                                Obligatory “correlation is not causation”.

                                                                                            3. 5

                                                                                              That is the problem right there. It.s not a “disagree” link, it’s a “flag” link. Obviously, looks like there are people abusing it because they disagree with something rather than unacceptable behaviour being reported.

                                                                                              Personally, I wouldn’t take it too seriously much less posting on other websites about it. But it is rather shitty to be told to take a chill pill just because one’s opinion is not widely accepted.

                                                                                              1. 1

                                                                                                Accounts with low “karma” can’t downvote but can flag. Perhaps this needs to be looked at..

                                                                                                1. 6

                                                                                                  There are no downvotes, only flags.

                                                                                                  1. 2

                                                                                                    TIL. I stand corrected

                                                                                                    1. 1

                                                                                                      Thanks for clarifying this. I was wondering where the downvote links were.

                                                                                                      1. 4

                                                                                                        iirc the downvotes were the flags, just with a downvote arrow instead of being hidden.

                                                                                                2. 3

                                                                                                  It doesn’t make sense to optimize for people who run away when told others disagree with them.

                                                                                                  I think discourse is improved when all participants feel safe. Then instead of spending energy in handling negativity emotionally, they can make better arguments. Moreover I don’t want to see valid viewpoints shut down due to mob shouting. I think human discourse works well with a variety of opinions.

                                                                                                  1. 16

                                                                                                    I think human discourse works well with a variety of opinions.

                                                                                                    I agree!

                                                                                                    I think discourse is improved when all participants feel safe.

                                                                                                    The problem we’re seeing in the last several years is that there are people who will never feel safe, at least not without massive sacrifices from everybody else in a community, often including systematic self-censorship and topic avoidance. The great online community question of the age is how to strike a balance between being welcoming and letting your demographics become taken over by–for lack of a better way of putting it–people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem.

                                                                                                    1. 9

                                                                                                      people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

                                                                                                      I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

                                                                                                      1. 15

                                                                                                        An (only slightly contrived) example:

                                                                                                        You run a message board that hosts tech content.

                                                                                                        On one extreme, you tell everybody “hey, we want you here! everybody is super nice and you’ll never be criticized!”.

                                                                                                        After some period of time, you end up with a bunch of users of C and a bunch of users of Rust.

                                                                                                        Some of those users cannot bear any perceived slight of their chosen language. There are C people that go off in a tizzy if anybody points out the memory safety issues in C, and there are Rust users that freak out if anybody complains about long compile times. The house rules–because remember, we’re optimizing for welcoming and that means supporting everybody, no matter how thin-skinned they are!–end up including “Never talk about shortcomings of another language.”

                                                                                                        Since engineering is all about tradeoffs, and since no language is actually good for all things, strict adherence to this rule ends up with only the most optimistic and vapid discussion–worse, it infects other topics. It’s really hard to explain why you might like immutability for concurrency if you can’t talk about memory safety issues because you’ll trigger the C programmers. It’s really hard to explain why you like dynamic or interpreted languages if you can’t talk about how much long compile times suck because you’ll trigger the Rust programmers.

                                                                                                        On the other extreme, you tell everyone “hey, we talk about tech and things on their own merits here, and if you can’t stand commentary that offends, fuck off!”.

                                                                                                        Our C users and Rust users come in.

                                                                                                        Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

                                                                                                        Obviously, there will be users who would be knowledgeable assets to this community but who are squicked out by this stuff and leave. There are other users who can put up with the rampant nonsense but who don’t want to spend the cycles required to weed through the bullshit to talk about what they want. Over time, the community stagnates unless it can find a way of attracting and retaining new users, and odds are that’ll be based on its reputation for no-holds-barred “discussion” rather than actual technical insight…this userbase will also skew towards younger folks who have a deficit of experience and all of the time in the world to share it. This is, incidentally, how /g/ works.

                                                                                                        ~

                                                                                                        I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

                                                                                                        My point isn’t to advocate for one or the other, but to point out that it’s a central question of community.

                                                                                                        It’s also a question whose answer shifts over time, as groups tweak the trajectory of their tribe.

                                                                                                        1. 8

                                                                                                          Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

                                                                                                          And there’s a pretty easy way to fix that – kick ’em out. There are communities which do that.

                                                                                                          Because as the saying goes, there are two ways to be a “10x programmer” and one of them is to have such an awful effect on people around you that they drop to 0.1x their prior productivity while you stay average. And even if someone really is “10x” (whatever that means) on some aspect of technical skill or knowledge, it’s still likely to be a net loss to put up with “asshole” behavior from them.

                                                                                                          1. 2

                                                                                                            This is good advice and knowledge for me as I try to build one or more Open Source communities. I think you are correct that such communities should have no room for such users.

                                                                                                            1. 1

                                                                                                              For the purposes of that thought experiment, we take the position in the second case where we do not eject members for being assholes.

                                                                                                          2. 3

                                                                                                            I saw another attempt at creating a HN clone go down a few years ago, not because of flaming or lack of interesting users but because some early users decided they and everyone else decided they needed to be much nicer than HN.

                                                                                                            I quit commenting soon after.

                                                                                                            You can check my comment history, I’m not at rude person but those rules made me feel unsafe. I started carefully considering each word. Then stopped posting,

                                                                                                            I wasn’t the only one. The “super nice” crowd didn’t have anything else to contribute and the site was down a few months later.

                                                                                                            1. 8

                                                                                                              Honestly, one of my main issues with HN has always been the focus on superficial civility. You can be an absolute horrendous awful drive-others-away-in-droves trainwreck of a poster there so long as you do it the “right” way. And you will be valued and rewarded for it.

                                                                                                              A real “anti-HN” would care less about those superficial aspects and more about dealing with the actual trolls and assholes.

                                                                                                          3. 4

                                                                                                            people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

                                                                                                            Perhaps I don’t see this as an issue. I believe, in my ideal community (where I’m stressing “ideal” because it’s probably not practically realizable), that any commenter with a good-faith argument should feel empowered to make that comment, even if others in the community viscerally disagree. There are two problems at hand here, one is defining what “good-faith” mean and the other is protecting arguments from being shouted out. When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements. In my mind, the easiest way to come to a consensus on what “good-faith” means is to clearly define it. This is why I’m a fan of Code of Conducts. They lay bare what a community considers “good-faith”, so the minimum bar of participation is met. Ideally the CoC would change infrequently and have community buy-in so that it acts as a minimum bar to entry.

                                                                                                            Ideally (again the ideal here not the practice, probably), we can create a community where there is no hostility after agreeing to “good-faith” rules. It’s one thing to enter heated discussion, but I’m hopeful that a community can use its own moderation tools to ensure that a heated discussion does not turn hostile.

                                                                                                            I just don’t see a world where diversity and hostility are equivalent.

                                                                                                            1. 1

                                                                                                              When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements.

                                                                                                              I make good-faith arguments all the time that seem to fall into the class of bad-faith arguments that @bitrot points out, so I get what you’re saying.

                                                                                                              I just don’t see a world where diversity and hostility are equivalent.

                                                                                                              In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄


                                                                                                              At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

                                                                                                              1. 2

                                                                                                                In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄

                                                                                                                Indeed, I am familiar with the Paradox of Tolerance. My hope is that a CoC or otherwise similar guiding document or constitution would provide a floor on tolerance, but everything beyond that would be tolerated. Not only does that set community baseline rules but it keeps rules distinct, clear, and constant. There’s a reason many real, representative governments go through due process to change laws.

                                                                                                                At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

                                                                                                                Perhaps the conclusion I should arrive to is that direct democratic (e.g. “upvote” and “downvote”) discussion sites just lead to dynamics that create and sustain echo chambers. I’m not sure, it’s certainly something I’m mulling about myself.

                                                                                                        2. 2

                                                                                                          when told others disagree with them.

                                                                                                          But that’s not what flagging tells you… If you’ve been doing everything alright, it tells you that others who disagree with you have leveraged a system unrelated to disagreement to make you look unreasonable, unkind, and/or trollish.

                                                                                                        3. 4

                                                                                                          I’d be interested in learning if mods have any way to detect brigading, possibly such as by looking at correlations in users who consistently flag the same comments.

                                                                                                          1. 13

                                                                                                            Yes, we do. If you look at the mod log you will see that people are banned for this practice now and then. Don’t do it.

                                                                                                            1. 1

                                                                                                              I’m not sure why you needed to admonish me like that at the end. I wasn’t probing your defenses. I was merely going to suggest this if you didn’t have it.

                                                                                                              No need to reply, I know this is a tough time for all the staff.

                                                                                                              1. 7

                                                                                                                Sorry, I should have been clearer that I wasn’t directing that at you in particular. I know you were trying to help.

                                                                                                        4. 8

                                                                                                          Preface: I don’t have his comments handy and am just looking at the system. Despite this I will be embarrassed if it turns out he was actually making an enormous mess 🤣

                                                                                                          Also want to add whoever worked on the existing system clearly put a lot of thought into it (e.g. factoring in flags on multiple stories, percent-flagged, and limiting to N users all seem wise, and thought went into the message). Probably no fun to see someone armchair quarterbacking with much less time thinking about the problem space than they have. Hope this at least crosses the threshold of possibly useful.

                                                                                                          The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.

                                                                                                          It looks like this is it. My guess is if you’re prolific, it’s inevitable you’ll pop over the hard numerical thresholds sometimes and the 10% of comments is the biggest barrier, followed by the top-30 ranking.

                                                                                                          10% of comments seems not that unlikely if 1) your recent comments are seen by lots of people (upvoted, early, etc.), 2) they address things a lot of people disagree about, and 3) some % of folks who disagree will flag, which seems likely as long as flagging is one of the easier ways to express disagreement (more below). You could do something like subtract the upvotes/N from the flag count (lots of fiddly options), just because there isn’t a number that represents visibility directly.

                                                                                                          The top-30 ranking has has the same considerations as flag count, and also if discussion is going well enough, the 30th “worst” person might not be that bad. (Somebody’s the least polite person in a group of absolute diplomats.) The histogram/rank data does seem interesting to expose to the more flagged folks. Given all that, I’m not entirely sure what I think about the wording of the message–definitely already phrased to aim for “stop and consider this” not “you’re a bad actor”, but I can also see how humans get defensive.

                                                                                                          People are really flag-happy, and this is what happens when you are.

                                                                                                          (Added clarifications here in [brackets] after friendlysock’s comment.)

                                                                                                          A nice thing about [the idea of downvotes for simple disagreement, as used on some other sites, is that it lets] people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on. I think the dream behind the current system is that not having [no-reason] downvotes leads to a very pluralist site where many people disagreeing (or fewer disagreeing strongly) doesn’t count against a comment as long as there’s no objective problem with the phrasing/on-topicness/etc. Realistically it seems as if without another outlet, some fraction of disagreement will get funneled into the flag system (even just through people looking hard for technically valid reasons to flag) even if you try to discourage it.

                                                                                                          Just as a random thought, you can offer downvotes without treating them as -1 upvote (-0.5 or -0.25 or whatever in a ranking algorithm), or disagree flags that you treat differently from others.

                                                                                                          tl;dr if I guessed at possible tweaks they might be allowing [no-reason] downvotes or disagree flags that have less impact than existing flags, just to keep that self-expression out of the flagging system; look for a way to factor in that being more-read will get your comments more negative attention; and maybe there’s something to do with the message wording, though it’s clearly already had a good amount of thought put in it.

                                                                                                          1. 16

                                                                                                            A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

                                                                                                            Yes, I very much agree with this. On Reddit and Hacker News, which lobster.s is modelled after, I think of as:

                                                                                                            • upvote: “I would like this to appear higher on the page”
                                                                                                            • downvote: “I would like this to appear lower on the page”

                                                                                                            Flag is a totally different thing than downvote – it means that the user is abusing the site, and continuing that behavior SHOULD result in getting banned from the site.

                                                                                                            But expressing minority opinions (thoughtfully) may result in downvotes (*), but it should NOT result in getting banned or encouraged to leave.


                                                                                                            As far as I can tell, the burntsushi warning was a result of mixing up these signals. I only read a portion of the Rust threads, so I haven’t followed everything, but from what I understand he was probably just expressing some opinions that people disagreed with, not actually abusing the site.

                                                                                                            So I think you have diagnosed it correctly – the lobste.rs UI is sort of mixing these 2 signals up by omitting downvotes. The “flag” button has 5 reasons, and “off topic” and “me too” should be a downvote, not a flag IMO.

                                                                                                            Again, the difference between the two is whether continuing the behavior should lead to a ban, and that’s a very important distinction.

                                                                                                            (Honestly I am surprised that I haven’t gotten this banner ever, given that there is a minority of people who disagree with my opinions on shell. Since I’m writing a new one, I’m opinionated about it, but also invite disagreement)


                                                                                                            (*) Some people may want opinions they disagree with to appear lower on the page, and some people might not. In my view it’s up to person to exercise that discretion – that’s why you get one vote :)

                                                                                                            1. 3

                                                                                                              upvote: “I would like this to appear higher on the page” downvote: “I would like this to appear lower on the page”

                                                                                                              That’s not actually what they mean on Reddit:

                                                                                                              Vote. If you think something contributes to conversation, upvote it. If you think it does not contribute to the subreddit it is posted in or is off-topic in a particular community, downvote it.

                                                                                                              ‘I would like this to appear higher/lower on the page’ is so ambiguous that it leads to different people misunderstanding it and applying it differently.

                                                                                                              1. 6

                                                                                                                I’d say that’s a prescriptive definition, but a descriptive one is more useful [1], and thinking about it that way will be more likely to solve the problem.

                                                                                                                That is, “users don’t read documentation”, and it means whatever the users think it means, and it changes over time. FWIW I’ve been using Reddit since 2007, and HN since 2011 or so, and there’s never complete agreement on what these things mean.

                                                                                                                I’ve seen the debate about downvote vs. disagree several times over the years on the sites. I choose to sidestep it with the vague definition – that’s a feature! :) It’s OK to be vague sometimes.

                                                                                                                In cases like this, where the number of flags leads to either banning or encouragement ot leave, I think it’s better not to be vague, and have 2 distinct mechanisms. This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

                                                                                                                [1] https://amyrey.web.unc.edu/classes/ling-101-online/tutorials/understanding-prescriptive-vs-descriptive-grammar/

                                                                                                                1. 2

                                                                                                                  This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

                                                                                                                  I don’t think it had both, rather the flag button just looked like a “downvote” button. You still selected from the same reasons that flag presented. This change is the one that altered the styling I believe. There’s plenty of places that refer to it as “voting” in the code still, but unless I am mistaken there was only ever the one mechanism.

                                                                                                            2. 7

                                                                                                              A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

                                                                                                              So, the purpose of downvotes was emphatically not to express disagreement…it was to signal one of a handful of error conditions (bad information, factual inaccuracy, etc.). There is very little utility in seeing how many people merely disagree about something, especially given how fast people are to be mean to each other.

                                                                                                              As a historical point, @jcs at one time disabled downvotes. It did not last more than a week or two before it was brought back.

                                                                                                              1. 11

                                                                                                                Sorry, I see how it was worded unclearly. I understand that the intention currently is to have no “disagree” button. I’m saying that having that button could reduce how often people flag comments they disagree with as having one of the error conditions, which they might do by by stretching definitions, picking a random reason, or just preferentially noticing real flaggable issues when they happen to dislike the content. (I think the unclarity comes from me using “downvotes” to mean “a no-reason downvote button” as opposed to flags as they exist–I edited to clarify above.)

                                                                                                                It may be useful to let people push a button for disagreement even if your algorithms assign a low weight to those clicks. I say low weight rather than zero because having a 100%-placebo button in the UI raises other issues, even if moderators give zero weight to an easy expression of disagreement. I’d probably give a no-reason-given downvote nonzero weight in my head but I don’t expect others to adopt my weights.

                                                                                                                Shorter, the idea is that people sometimes find a way to “downvote” for disagreement however you set up your UI, and you might get a net harm reduction by just giving them the button they want and then deciding what to do with the data.

                                                                                                                1. 6

                                                                                                                  Ah, so kind of like the buttons on crosswalks?

                                                                                                                  Anyways, thank you for elaborating further. :)

                                                                                                                  1. 4

                                                                                                                    Very well said. Even if the system is not equipped with a “disagree” button, some users may desire one and use whatever means are at hand to express themselves. The phrase, “the purpose of a system is what it does,” comes to mind.

                                                                                                                    If existing mechanisms are not meant for use in disagreements, then we might ask if that is actually how they are being used.

                                                                                                                  2. 4

                                                                                                                    That’s nice, but you can’t make users behave the way you want them to behave, if you’re not going to actively act against it. People not only downvote because they disagree, but they’ll also flag things as spam or off-topic or “troll” just because they disagree, or feel you’ve insulted their particular religious (as in vimacs-vs-spaces) beliefs.

                                                                                                                    1. 6

                                                                                                                      I think I’d prefer a system that discourages misuse of flags. Maybe a similar message for people who flag a lot

                                                                                                                      1. 3

                                                                                                                        Sure, folks determined to game things would still flag. My theory is decently many who flag today are not determined to game things, and would click a down arrow for “just no” given the option. That’d reduce noise in the flag system even though it wouldn’t eliminate it. (And not saying you couldn’t do other changes too.)

                                                                                                                    2. 4

                                                                                                                      in fact, you can have placebo downvotes that show up for the person doing the downvoting but do not affect the vote count otherwise. that would keep (almost) everyone happy, i think.

                                                                                                                    3. 6

                                                                                                                      Agreed.

                                                                                                                      I think it’s fair to discuss whether the wording of the warning message can be improved, but various comments here that pretty much amount to “person X is popular, he should be exempt from the rules” are just tiring.

                                                                                                                      1. 8

                                                                                                                        How is being flagged a rule? It’s not a rule. That is the whole problem.

                                                                                                                        Anyone can flag any comment any time.

                                                                                                                        1. 2

                                                                                                                          Receive too many flags, get a notice. That’s the rule.

                                                                                                                          1. 9

                                                                                                                            That is not a rule in the sense of rule that can be followed by users. I did’t read a single comment suggesting that shouldn’t apply to some user because of its popularity, but rather many people pointing out that this is a clear case illustrating that such system is broken and should be fixed for all users.

                                                                                                                      2. 5

                                                                                                                        rather than continue to engage in a community they don’t fit into

                                                                                                                        Is lobster a community that accepts a plurality of opinion? If it is, then you will always have people disagreeing with you and therefore be liable to be reported by some of these people.

                                                                                                                        Democracy leads to lynching. Mass reporting is basically the lynching of internet persona. Any kind of automatic penalty that can be initiated via reporting is simply giving power to the angry mob. Autoban and autosuspend based on reports have always been abused.

                                                                                                                        1. 3

                                                                                                                          You’re still here, so evidently the suggestion doesn’t apply to you. You’re glad to contribute.

                                                                                                                          Why not show the warning to flag-happy people? I don’t have stats but I feel like that could be a better indicator of who doesn’t want to be here.

                                                                                                                        2. 21

                                                                                                                          Why do we need this passive aggressive banner anyway? It looks bad for Lobster, not for the person seeing it, and it’s not clear what problem it’s supposed to solve.

                                                                                                                          1. 19

                                                                                                                            We consistently see the vast majority of bad behavior come from the same few users. There’s a phenomenon, called the normalization of deviance, whereby people whose behavior is far away from the norm often genuinely don’t realize just how far it is. The banner, and the histogram that it links to, are intended as a way for people to self-correct. I personally am a big believer in giving people opportunities for redemption, and the histogram makes the point in a way that mere words can’t.

                                                                                                                            1. 6

                                                                                                                              far away from the norm

                                                                                                                              Far away from the norm doesn’t mean bad. You seem to be assuming that it is.

                                                                                                                              1. 8

                                                                                                                                I appreciate you raising that. On the contrary, I agree that sometimes people’s behavior is far from the norm for positive reasons. I do think that, even in those cases, everyone can benefit from a heads-up and a chance to reflect about whether it’s what they want.

                                                                                                                                1. 3

                                                                                                                                  As in reflecting on do you want to be good when everybody else is bad?? Am I reading that right?

                                                                                                                                  1. 7

                                                                                                                                    Well, yeah. I know for a fact that people have left Lobsters for that reason (or at least, that’s how they see it). While I am sad when people leave, I don’t think anyone is obligated to stay somewhere that doesn’t make them happy. It would be wrong of me to try to trick or coerce people into staying when their free choice, with all the facts available, would be to leave.

                                                                                                                                    I’m not really sure why you’re asking this, though. The flagging system doesn’t have the ability to determine who’s right and who’s wrong. Even if I wanted to take my personal concept of “good” and write code that ignores flags that aren’t “good”, there is no existing technology that can do anything like that. If that’s what you’re advocating for, feature request noted but I’m not able to do it.

                                                                                                                                    1. 3

                                                                                                                                      I’m not really sure why you’re asking this, though

                                                                                                                                      I am not asking anything. I was commenting on an implicit bad argument you made (the implication that the norm is correct). You decided to double down by arguing that even if somebody was correct, they should ‘reflect about whether it’s what they want’. I simply then pointed out that this doesnt reinforce your original argument, because it itself is bad, because having a bad feature (aggressive warning about being reported) is not validated by having that feature also be useful to do useless thing (letting people reflect on whether they should be good, because the answer is always yes).

                                                                                                                                      I don’t think anyone is obligated to stay somewhere that doesn’t make them happy.

                                                                                                                                      Of course not. The point is that assuming that lobsters want a community that engage in meaningful discussion, then it should make people want to stay and be happy, that is a worthy goal. all of this is following your argument for the banner, the argument which I consider to be invalid, as pointed out.

                                                                                                                                      1. 5

                                                                                                                                        Being distant from the norms can hold little moral judgment and yet be relevant when operating in a constructed community.

                                                                                                                                        1. 2

                                                                                                                                          Okay, I guess that’s fair.

                                                                                                                                          I think we disagree about something in there, but it sounds like figuring out what isn’t a high priority for either of us? I appreciate the discussion, anyway.

                                                                                                                                  2. 1

                                                                                                                                    I at least didn’t read that implication into the post. Not all ways of being far from the norm are likely to result in seeing the banner. It isn’t a perfect system but it’s definitely not symmetric, so that assumption isn’t necessarily required.

                                                                                                                                  3. 2

                                                                                                                                    Thanks for clarifying. But I still think it doesn’t make Lobster looks good because this feature implies that “deviant” behaviour should be corrected and can be automatically detected, and I expect most people (especially in tech) aren’t comfortable with either of these.

                                                                                                                                    I’m wondering what’s the worse that could happen if the banner is not displayed? Maybe the person who’s being flagged all the time will get the point and adjust their behaviour, or maybe they don’t need to adjust anything and it’s other people that will start seeing that perhaps that person has a point. I also believe in giving people a second chance but I don’t think the solution to everything has to be algorithms, sometimes simple human interactions are preferable.

                                                                                                                                    1. 4

                                                                                                                                      The hypothesis that motivates the banner and histogram is that, by displaying them, those conversations about what to adjust will happen sooner and be more productive. The goal is to head off situations where somebody doesn’t believe mods when we tell them something needs to change, and they ultimately leave the site. It may be right or wrong but that’s the motivation.

                                                                                                                                    2. 1

                                                                                                                                      Do you think that the banner has an observable positive effect? Do you see a change in behavior from the few users you say produce the majority of bad behavior? As friendlysock says above, they see the banner nearly all the time. Do you think they have been falsely-flagged by this system? If so, can you estimate how high the false-positive rate of showing this banner? If friendlysock has been appropriately warned by this system, have you seen any change in their behavior onsite?

                                                                                                                                      If it is truly only a few users producing undesirable behavior, could it be more effective to simply reach out to those users personally?

                                                                                                                                      1. 4

                                                                                                                                        I’m not going to publicly discuss my personal feelings about specific community members. It wouldn’t be fair to anyone.

                                                                                                                                        I don’t have a methodology to conclusively determine how much influence the banner has had, but the site has a lot less flagging in general than it did a couple years ago. I take that to be an indicator that things have calmed down significantly, and I do think that’s good.

                                                                                                                                        I do reach out to people personally from time to time, as do the other mods. The banner and histogram were created to address a shortcoming in that approach, where sometimes people aren’t willing to listen to criticism unless they’re presented with data that validates it. I’m skeptical of going fully data-driven for everything, but I think it’s nice to have at least this one particular metric for those who find it easier to take concerns seriously when there’s a metric.

                                                                                                                                  4. 20

                                                                                                                                    Additional suggestion to all the previous improvements: make that banner a thing that’s moderator-initiated. I of course don’t have statistics on how many people see it (I never got it, so I just learned about it now, anecdotally) but I could imagine that this is rare enough that moderators could just evaluate sending this message to users by hand after the system flags the user internally.

                                                                                                                                    1. 6

                                                                                                                                      I of course don’t have statistics on how many people see it

                                                                                                                                      I don’t have that statistic either, but as I posted elsewhere, only 13 people have been flagged more than me in the last month and I haven’t seen it. Only 7 people have been flagged 10 or more times. I’d therefore expect that trimming it down to 5 people that moderators need to take a look at per month is quite feasible.

                                                                                                                                      If even that is too much of a load for the moderators, then it might be possible to select a dozen people who have been active recently but not participated in any of the flagged threads to do a triaging pass. At that point, having someone reach out to the person and say ‘you’ve had a lot of things flagged recently, is there anything going on in your life that you want to talk about?’ would be a fairly low load.

                                                                                                                                      1. 3

                                                                                                                                        You can a chart that gives you a clear idea how many people see it at /u/you/standing. So you can see it at /u/david_chisnall/standing and I can see it at /u/hoistbypetard/standing. The red Xs should represent users seeing the warning.

                                                                                                                                        I’d never suggest that someone else should just add a thing to their plate, but it does seem like the kind of thing where, if there were a few active moderators, requiring someone to approve the red box might be an improvement for a relatively low effort. That feels like the kind of statement I should only make if I’m accompanying it with a patch, though, and I can’t offer that right now.

                                                                                                                                    2. 15

                                                                                                                                      Yep, I think the current wording will probably trigger exactly the wrong people “have I overstepped a border? Probably” and the people who are actively engaging in a little bit of flaming won’t be bothered.

                                                                                                                                      1. 7

                                                                                                                                        Following on the “let’s get shit done” vibe, is there an open issue for discussing this in more detail on github? I did a quick lookup and didn’t find any, would be cool if it was posted here (even better if as a top comment)

                                                                                                                                        1. 5

                                                                                                                                          Good idea, thanks! (Obvious in retrospect, like many good ideas.) I created an issue.

                                                                                                                                        2. 4

                                                                                                                                          I mildly agree with both of those things, but I also think that the suggestion further down the thread to re-introduce downvoting as “non-bannable feedback” could address the issue more directly.

                                                                                                                                          https://lobste.rs/s/zp4ofg/lobster_burntsushi_has_left_site#c_padx5h

                                                                                                                                          Some people may misuse flags and downvotes, but overall I think these guidelines are intuitive and easy to remember:

                                                                                                                                          • upvote: this should go higher on the page
                                                                                                                                          • downvote: this should go lower on the page
                                                                                                                                          • flag: if the user continues this behavior, they should eventually be banned
                                                                                                                                          1. 2

                                                                                                                                            In my post I’m trying to focus on an easy thing to do to help reduce this problem. I have the impression that rewording the banner is something that’s reasonably easy, while “let’s reintroduce downvotes” or “the banner should be approved by moderators before being shown” are important change in process that require a lot of discussion. They may be excellent ideas, but it’s a lot more work to discuss and turn them into action, so I think it’s best not to mix the two kind of suggestions if we want to change anything at all in the short term; they should be discussed separately, and maybe differently. (I could just send a PR to rephrase the banner; any of the other suggestions requires more discussion with contributors, etc.).

                                                                                                                                            1. 1

                                                                                                                                              Yes, it makes sense to do the easy things first. My impression was that we had downvotes several months ago, so it shouldn’t that big a change to restore them, but I’m not involved so I could be wrong.

                                                                                                                                              I was mystified by the removal of downvotes, and never saw the rationale for it. I saw one thread discussing it 3 or 6 months after they were removed, which still didn’t answer my questions about why they were removed. It made the site worse for me, but not enough to really complain about it. I know this is an all-volunteer effort and I didn’t have time to look at it myself.

                                                                                                                                        1. 9

                                                                                                                                          Signals are the most exciting part of the POSIX world. This post missed out my favourite bit though: signal handlers also receive a ucontext_t, which contains a complete register dump of the thread that was interrupted by the signal. You can use this for all sorts of exciting things. I’ve written code that handles a signal on system calls where the kernel does not allow the syscall, extracts the arguments, and then does an RPC to a more privileged process to ask it to perform the system call based on a dynamic policy, then injects the success / failure return value back into the signal’s ucontext_t. Because this is C++ and not C, it’s all driven by template instantiation that figures out the registers to pull out based on the argument types of the function that does the RPC. I imagine you could probably do the same thing in Rust.

                                                                                                                                          1. 4

                                                                                                                                            Wow, I was unaware the ucontext_t was usefully mutable… that’s a little terrifying.

                                                                                                                                            1. 2

                                                                                                                                              For a joke I once wrote a signal handler that increments the instruction pointer on segfault and illegal instruction signals / SIGSEGV and SIGILL. https://github.com/RichardBarrell/snippets/blob/master/no_crash_kthxbai.c It sometimes even works.

                                                                                                                                              You can also implement less-silly things with SIGSEGV handlers, like I think at least one clustering system might have used SIGSEGV handlers to present the illusion of shared memory across a network in userspace.

                                                                                                                                              1. 2

                                                                                                                                                I think at least one clustering system might have used SIGSEGV handlers to present the illusion of shared memory across a network in userspace

                                                                                                                                                The TreadMarks system (pdf) from Rice University in the late ‘90s did that (see p. 15 for a description). At least one commercial product was based on that scheme, Intel’s Cluster OpenMP (I think now discontinued). Here’s an old Intel whitepaper (also pdf), with a description of the mechanism on pp. 5-6.

                                                                                                                                                1. 2

                                                                                                                                                  I’ve thing I don’t understand about these schemes: how did they handle memory pressure? If I have a page in the application that’s just a read-only mirror of a page from another machine in the cluster, it should be okay to drop it at any time, just like how it’s okay to drop any unmodified page of memory containing filesystem cache. You just need the page to generate another segv if it’s accessed again after being dropped. But from the kernel point of view, it’s application data that can’t be safely stored e at all.

                                                                                                                                                  It doesn’t look like there’s any way to tell Linux that pages can be unmapped safely with madvise(). All the options I see in the man page (like MADV_DONTNEED) give you zero filled pages rather than segfaults if accessed again after being dropped.

                                                                                                                                                  1. 2

                                                                                                                                                    You can mmap anonymous memory over a page to discard it forcibly. The portable idiom is to map anonymous memory with PROT_NONE over the region. You’ll then get a trap. For a distributed shared memory thing, you first map the page PROT_READ, then ensure any modifications are written to other machine(s), then map PROT_NONE to discard it. This gives you sufficient to implement MESI:

                                                                                                                                                    • Initially pages are PROT_NONE (Invalid state).
                                                                                                                                                    • When you take a SEGV, fetch the data and insert it with PROT_READ[1] (shared state).
                                                                                                                                                    • When you take another SEGV, broadcast invalidate the page on all other machines and mark it PROT_READ | PROT_WRITE locally (briefly exclusive state, exclusive one instruction after the signal handler returns).
                                                                                                                                                    • After a while (or after you receive messages from other nodes trying to put it into shared state), mark it PROT_READ and send the data to other requesters (back to shared state).
                                                                                                                                                    • If you are in shared state and receive an invalidate, mark it PROT_NONE (ideally mapping a new anonymous page PROT_NONE to free up physical memory) and transition back to invalid state.

                                                                                                                                                    False sharing is always a problem with this kind of system. If two nodes are writing to objects that happen to be on the same page then they will keep flipping the entire page between modified and invalid state and become really slow. Most multithreaded software tries to avoid false sharing at the cache-line granularity but avoiding it at the page granularity is much harder.

                                                                                                                                                    For general-purpose use, these things also suffer from reliability issues. If you’re running a 512-node cluster with a distributed shared memory system, you are 512 times more likely to encounter a catastrophic failure than if you have one node of that cluster. Most modern distributed systems try to explicitly reason about failure and anything that makes sharing transparent makes this impossible.

                                                                                                                                                    [1] To do this atomically, you need to create an anonymous memory object (or a[n unlinked] file if your OS doesn’t support anonymous memory objects), mmap it read-write somewhere, populate it, and then map it read-only in its destination, at which point you can discard the read-write mapping if you want.

                                                                                                                                                    1. 2

                                                                                                                                                      The reason I’m wondering about madvise is that this scheme you’ve doesn’t give the kernel the ability to (unilaterally) discard pages for which the program has a read-only copy that it knows it isn’t the sole holder of, in response to memory pressure.

                                                                                                                                                      The application has to voluntarily give them up, and it may not know that it needs to (and memory pressure may have put the system into a state where the kernel doesn’t want to or can’t schedule the application).

                                                                                                                                                      I know you can implement mesi in userspace, what I’m wondering is can you implement swap

                                                                                                                                                      1. 2

                                                                                                                                                        The reason I’m wondering about madvise is that this scheme you’ve doesn’t give the kernel the ability to (unilaterally) discard pages for which the program has a read-only copy that it knows it isn’t the sole holder of, in response to memory pressure.

                                                                                                                                                        There are similar mechanisms. MADV_FREE lets the kernel discard the page and replace it with a page of zeroes. You want a scheme that lets the kernel discard the page and replace it with a no-access mapping that you could then replace. I think XNU has a mechanism like this that iOS uses for some cache infrastructure so that the OS can discard pages that the application can recreate.

                                                                                                                                                        I know you can implement mesi in userspace, what I’m wondering is can you implement swap

                                                                                                                                                        Yes - I did almost 20 years ago and I wasn’t the first one. There are three aspects to userspace swapping:

                                                                                                                                                        • Can you discard pages and get a trap on access that you can fix up?
                                                                                                                                                        • Can you remove and replace a page without doing anything that would cause a page fault?
                                                                                                                                                        • Can you get useful triggers that tell you when you need to swap?

                                                                                                                                                        Of the three, the first is fairly easy with POSIX, the second is difficult but possible, the third is impossible in portable code. Android and XNU have mechanisms for delivering various low-memory notifications to userspace, upstream Linux and other *BSDs don’t. Windows does, but it isn’t delivered at a useful time (it’s fairly easy to get into a situation where the kernel refuses to allocate more memory for userspace but the low-memory notification isn’t delivered). You can have some per-process threshold after which you start swapping, but memory pressure is a global property and so local optimisation is not a good solution.

                                                                                                                                                        1. 1

                                                                                                                                                          There are similar mechanisms. MADV_FREE lets the kernel discard the page and replace it with a page of zeroes

                                                                                                                                                          I was discussing that & why it’s unsuitable in the post you replied to above? This comes off as slightly rude. :(

                                                                                                                                                          I did almost 20 years ago and I wasn’t the first one

                                                                                                                                                          Nice! Which mechanism did you use to get pages discarded under low memory conditions?

                                                                                                                                                          Can you get useful triggers that tell you when you need to swap?

                                                                                                                                                          I assume in practice people must have tuned userspace swap systems to start discarding pages long before getting close to a hard OOM, since the kernel might need to allocate pages in order to be able to deliver a notification to & schedule the userspace process that is holding onto the memory.

                                                                                                                                                          1. 2

                                                                                                                                                            There are similar mechanisms. MADV_FREE lets the kernel discard the page and replace it with a page of zeroes

                                                                                                                                                            I was discussing that & why it’s unsuitable in the post you replied to above? This comes off as slightly rude. :(

                                                                                                                                                            Sorry, I didn’t meant to imply that you weren’t aware of this mechanism but to highlight that the change from MADV_FREE to what you want is fairly small. The implementation of MADV_FREE has to change the permission from read-write to read-only. Changing that to no-access instead and not marking the new page as CoW would be quite a small kernel change.

                                                                                                                                                            Unfortunately, signals are really bad for composition. You really want the OS to have a different signal handler for each memory object. Windows has a mechanism something like this. Otherwise, you need to rely on every signal handler invoking the one that was returned in the sigaction call if it can’t handle it.

                                                                                                                                                            I did almost 20 years ago and I wasn’t the first one

                                                                                                                                                            Nice! Which mechanism did you use to get pages discarded under low memory conditions?

                                                                                                                                                            I used a per-process in-core limit. I did hack in something to FreeBSD to expose to userspace one of the thresholds that’s used to poke the pager via a signal, but it didn’t work well. I also played a bit with L4 Hurd, which had some really nice mechanisms for doing this, but I was only able to run it in a VM.

                                                                                                                                                            Can you get useful triggers that tell you when you need to swap?

                                                                                                                                                            I assume in practice people must have tuned userspace swap systems to start discarding pages long before getting close to a hard OOM, since the kernel might need to allocate pages in order to be able to deliver a notification to & schedule the userspace process that is holding onto the memory.

                                                                                                                                                            On XNU, as I recall, there are three different levels of trigger. There’s the polite ‘please delete caches’ notification, there’s an aggressive ‘the OOM killer will kick in soon’ and then there’s the OOM killer (which starts by killing processes that have opted in to sudden termination.

                                                                                                                                                            Part of the problem here is thundering herds: in the kernel, there’s typically a single pager (maybe a small number for different types of memory). In userspace, any number of processes can register for low-memory notifications. If they all need to allocate a small amount of memory to free a large amount and they all wake up at once, they end up making everything much worse.

                                                                                                                                                            1. 1

                                                                                                                                                              Unfortunately, signals are really bad for composition. You really want the OS to have a different signal handler for each memory object. Windows has a mechanism something like this. Otherwise, you need to rely on every signal handler invoking the one that was returned in the sigaction call if it can’t handle it.

                                                                                                                                                              Agreed. It’s the same problem as any kind of global state like cwd, I guess.

                                                                                                                                                              I’m of the opinion that catchable asynchronous signals are an unfortunate historical design full stop. ;)

                                                                                                                                                              You could make up a convention in user space where e.g. everyone uses a single library that maintains a hash table, but meh.

                                                                                                                                                              AIUI segfaults are so slow, it’s likely that anyone who cares enough to benchmark would rather use a btree/hash and do the lazy computation with explicit function calls.

                                                                                                                                                              I mean how often are we going to do “get everyone to agree on a convention for signal handlers” but not “get the like three programmers who access this array to call get_it(i) instead of array[i]”?

                                                                                                                                                              In userspace, any number of processes can register for low-memory notifications. If they all need to allocate a small amount of memory to free a large amount and they all wake up at once, they end up making everything much worse

                                                                                                                                                              Oof.

                                                                                                                                                              I wonder if the kernel could (or maybe does?) mitigate this by delivering the low memory notifications to each process in decreasing order of their memory usage, leaving say a dozen milliseconds between?

                                                                                                                                                              (And keep a flag for “has been sent a low memory notification” in the process info struct so this doesn’t need a dynamic memory allocation to avoid notifying any process repeatedly when the order changes while the list is being traversed. 😅)

                                                                                                                                                              I used a per-process in-core limit.

                                                                                                                                                              In all honesty this is probably the best / most predictable solution for a lot of workloads. e.g. with MySQL you tell it up front how much memory to use, so you pick a number based on how much you have and how much slack you need left for other stuff on the same box. I’ve never met anyone whose day was ruined by this scheme.

                                                                                                                                                2. 2

                                                                                                                                                  You can also implement less-silly things with SIGSEGV handlers, like I think at least one clustering system might have used SIGSEGV handlers to present the illusion of shared memory across a network in userspace.

                                                                                                                                                  Yeah that sort of thing seems “normal”, the fiddling with the ucontext_t just by… fiddling with it, with no need to do something like setcontext/swapcontext or longjmp is what caught me off guard.

                                                                                                                                                  1. 2

                                                                                                                                                    It is weird but kind of makes sense: in the signal handler context, the thread is stopped and its register state has been stashed somewhere in memory, so you can just fiddle that memory like any other data structure.

                                                                                                                                                    Whereas in normal contexts the register state is in the registers that your C code is using, so you need something like setjmp that’s written in assembly and/or is a built-in.

                                                                                                                                                    1. 1

                                                                                                                                                      Yeah, D has an option to enable this too. It isn’t done by default though because it has its weird platform quirks and running it in a debugger is frequently more useful anyway.

                                                                                                                                                      Another experiment the D people did was to mprotect things while garbage collecting, then do a kind of userspace page in on the segv signal. Turned out slower than just using the pause all threads global lock approach though, but it was still kinda cool being able to do a job typically done by the kernel.

                                                                                                                                                      1. 1

                                                                                                                                                        IME most complex runtimes (i.e ones that have a JIT) do. Mono for instance does. It turns out to be a problem when I was porting it to AIX, because AIX doesn’t have an unmapped null page - you can just….read zeroes back from null. AFAIK this was either a compat or a performance thing. Instead, I had to turn on the usually debug-only emitted-instructions null checks for the platform.

                                                                                                                                                        1. 1

                                                                                                                                                          There’s nothing in .NET that requires that null have a zero bit pattern. I wonder how much harder it would have been to keep this JIT mode, map a page somewhere in the address space as no-access and then use its address as the null pointer representation. I guess you get quite a bit of speedup from the fact that compare-against-0 is a heavily optimised instruction in most CPUs, so this may end up being slower.

                                                                                                                                                          1. 1

                                                                                                                                                            I think keeping null == 0 is probably easier for FFI/mapping to HW better, yeah.

                                                                                                                                                      2. 1

                                                                                                                                                        I’ve actually done that, not as a joke. This code is in the CHERI test suite, which checks that you get traps for the operations that you expect. The individual tests can try to do a load or store that should fault and then check if a counter is incremented by the signal handler. The same tests can also run bare-metal with the equivalent logic in the interrupt service routine, but when running on an OS the kernel delivers a signal to userspace in response to the trap.

                                                                                                                                                        During my PhD, I wrote some code for out-of-core data management that caught SIGSEGV, fetched the data and populated the faulting page. That never went anywhere because it was noticeably slower than other approaches.

                                                                                                                                                        1. 1

                                                                                                                                                          context->mc_pc += 4;

                                                                                                                                                          Ah, bless that simple uncomplicated RISC instruction encoding scheme. ❤️ I was wondering where the other 99% of the decoding code must be until I looked up and saw that it is MIPS!

                                                                                                                                                          During my PhD, I wrote some code for out-of-core data management that caught SIGSEGV, fetched the data and populated the faulting page.

                                                                                                                                                          There’s a Linux thing I remember hearing of that’s supposed to use the hypervisor support to let an “ordinary” process use the hardware page fault mechanism directly, and I think it makes applications that use faults for implementing semantics several times faster. But I’ve forgotten the name of it and damned if I can find it again.

                                                                                                                                                          1. 2

                                                                                                                                                            context->mc_pc += 4;

                                                                                                                                                            Ah, bless that simple uncomplicated RISC instruction encoding scheme. ❤️ I was wondering where the other 99% of the decoding code must be until I looked up and saw that it is MIPS!

                                                                                                                                                            The decoder is simple. The logic for branch delay slots is fun. If you take the exception in a branch delay slot, you need to identify that you’re in the delay slot and then advance the PC to the instruction that’s the branch target. Fortunately, MIPS has only one delay slot and so if the instruction in the delay slot traps, it won’t have been able to modify any of the registers that the branch instruction used.

                                                                                                                                                            During my PhD, I wrote some code for out-of-core data management that caught SIGSEGV, fetched the data and populated the faulting page.

                                                                                                                                                            There’s a Linux thing I remember hearing of that’s supposed to use the hypervisor support to let an “ordinary” process use the hardware page fault mechanism directly, and I think it makes applications that use faults for implementing semantics several times faster. But I’ve forgotten the name of it and damned if I can find it again.

                                                                                                                                                            I saw some references to that in the code a couple of weeks ago, but then couldn’t think of sufficiently similar search terms to find it when I wrote this. If anyone knows what it’s called, please reply here!

                                                                                                                                                            1. 1

                                                                                                                                                              The logic for branch delay slots is fun.

                                                                                                                                                              Honestly that doesn’t take much away from the elegance of it, you only have a handful of lines for handling the BD case there. :)

                                                                                                                                                  1. 39

                                                                                                                                                    Zig is a language with manual memory management and no runtime, like C.

                                                                                                                                                    This is your obligatory reminder that C does in fact have a runtime. C is just in the enviable position that its runtime has been folded into the Unix operating system API. On an OS that is not as C-centric, this distinction is more visible.

                                                                                                                                                    1. 7

                                                                                                                                                      Even on a unixalike crt0 is there, lurking, doing things like populating environ and calling main.

                                                                                                                                                      1. 9

                                                                                                                                                        The original framework - don’t call us, we’ll call you.

                                                                                                                                                        1. 4

                                                                                                                                                          Have you heard about the popular DI framework LD?

                                                                                                                                                      2. 5

                                                                                                                                                        Yup anyone remember msvcrt.dll ? (presumably this still exists, but thankfully I haven’t seen it in ages)

                                                                                                                                                        1. 1

                                                                                                                                                          It’s Complicated™ as MS tends to alternate between versioned CRT and universal unversioned CRT - that, and you can statically link it, but unlike a Unix, it doesn’t matter because the system libraries don’t involve the CRT.

                                                                                                                                                        2. 5

                                                                                                                                                          In fairness, c can be used without a runtime

                                                                                                                                                          …but so can many other languages.

                                                                                                                                                          (I’m not counting libgcc; it’s an implementation detail.)

                                                                                                                                                          1. 2

                                                                                                                                                            …a few other languages, at least.

                                                                                                                                                        1. 1

                                                                                                                                                          I use a debugger to do printf debugging. It’s nice. You can start with an interactive break point and upgrade to printing. You can print things like registers and stack traces that can be harder/more awkward to do in code. You can toggle particular prints on and off at runtime. You can trace calls to a function etc. You can send the output to a separate log. I very rarely single step interactively. You can do this with gdb or windbg, I haven’t used lldb in anger but imagine it is programmable as well.

                                                                                                                                                          Automated debugging is a powerful technique, and the automation means you can more safely use it in production, it’s not responding at human scale.

                                                                                                                                                          1. 2

                                                                                                                                                            Do you consider logging to be a form of automated debugging? I’ve found that adding what many feel to be “too much” logging to production systems has made solving and repairing many bugs much easier.

                                                                                                                                                          1. 3

                                                                                                                                                            Gonna open my mouth and remove all doubt… what’s the joke?

                                                                                                                                                            1. 5

                                                                                                                                                              I think this was just mistagged as satire and have corrected it. Though it’ll be pretty embarrassing if it also went over my head, what with taking my handle from x86 assembly.

                                                                                                                                                              1. 1

                                                                                                                                                                See the sibling. I’m all but certain it was intended as satire.

                                                                                                                                                              2. 1

                                                                                                                                                                Assembly programming didn’t stay in the 50s, it evolved along with high-level languages incorporating structural, functional, and objective-oriented programming elements. It plays well with modern APIs and DOMs

                                                                                                                                                                I would be very surprised if this weren’t a joke.

                                                                                                                                                                1. 2

                                                                                                                                                                  This could be tongue-in-cheek, but the bulk of the article looks serious.

                                                                                                                                                                  1. 2

                                                                                                                                                                    Judging by the rest of the site it is not a joke.

                                                                                                                                                                1. 7

                                                                                                                                                                  Something that surprised me was how similar V8’s object representation was to Self’s object representation. Perhaps the authors were influenced by the paper, or perhaps they simply came up with a similar representation on their own.

                                                                                                                                                                  Lars Bak worked on both Self and V8 so that is likely how the influence spread.

                                                                                                                                                                  1. 2

                                                                                                                                                                    That’s a really nice catch! That’s quite interesting indeed, thank you.

                                                                                                                                                                    1. 3

                                                                                                                                                                      Urs Hölzle was a key member of the Self team and hired Lars. Dave Ungar was last seen working on Swift at Apple.