1. 5

    Very cool! Works well. Gonna use this for a local sneakernet I’m running. 50GB of data encrypted in 3.2 minutes.

    Do you think this code could benefit from parallelization? Make it even faster?

    Also, any plans to publish to crates.io? It’d be really handy if I could just say cargo install bitbottle

    1. 6

      I’ll add “figure out crates.io” to my to-do list. :) I’m not sure if parallelization would help; the bottleneck seems to be disk I/O or LZMA2, and both are serial. I’m also worried about the complexity cost of adding concurrency, unless it makes a huge difference.

      1. 4

        the bottleneck seems to be disk I/O or LZMA2, and both are serial. I’m also worried about the complexity cost of adding concurrency, unless it makes a huge difference.

        For what it’s worth, the zstd library includes multithreaded compression (and Rust bindings exist).

        It and brotli also fill the space of “denser than Snappy, but faster than LZMA2” and have faster decompression than LZMA2 even at their denser settings. zstd started from the fast side (it’s from Yann Collet, LZ4’s author) and brotli started from the dense side (one of the authors is Jyrki Alakuijala, a Google compression specialist), but both offer a wide range of speed/density tradeoffs now.

        1. 1

          Ye, I’d look into zstd or lz4 instead of lzma2.

      1. 3

        I love the note that a multiply and shift mostly fixes the initial ‘bad’ hash: if the problem is entropy badly distributed throughout the word, a multiply can quickly mix the all the bits into the upper few. I suspect some inherited wisdom about designing a non-crypto hash dates back to when it was much more expensive, relatively, to multiply by an arbitrary 64-bit number in a hot loop. (xxhash, though not made for hashtables, hashes at 19.4GB/s on Intel Coffee Lake with 64-bit multiplies and no SIMD.)

        I tend towards “just use something SipHash-y” for roughly the reason I like ciphers as PRNGs: they’re fast enough and get rid of a whole class of potential problem. Maybe the argument is weaker in hashtables, because more apps really want accesses to be fast, and there are other ways at the hash-flooding/collision problem. Still, hashes are at least an area where tuning too much for the wrong thing can end up penny-wise and pound-foolish.

        1. 4

          As browsers expose more APIs, exposing the Unicode data they already need for their own regexp implementations, sorting, etc. seems like a reasonable thing to want. (There are already substantial i18n APIs.)

          Many wasm programs need to process text and all need the same data to do so, and sometimes following the browser’s idea of Unicode may be an added benefit–gets an old program ‘free’ Unicode version updates, or conversely helps grapheme clustering match the browser’s when the browser is not up to date.

          1. 2

            (Author here) I would love this. I think that even just improving the existing Intl.Collator API could do good enough. The problem with it currently, from what I have seen, is that it’s actual behavior is not formally specified. That is, it works well if you’re, say, trying to sort strings in the user’s locale - which is what the API was designed to do - but if you expect those results to be consistent across browsers you’re out of luck and will quickly run into issues like[0]

            If the behavior was consistent across browsers, I think that’d be good enough to rely on the Intl APIs in general. But yeah, differing behavior for say a regexp engine across browsers is.. well.. not desirable, to say the least.

            [0] https://stackoverflow.com/questions/33919257/sorting-strings-with-punctuation-using-intl-collator-is-inconsistent-across-brow

            1. 1

              Oh, wow, Collator being almost right is kind of a maddening snatching-defeat-from-the-jaws-of-victory situation.

              Zorex looks cool! A regex-like language that does things Everyone Knows Regexes Can’t Do seems like it could be useful in a lot of places.

          1. 10

            Clever trick, but I think that down this path lies madness. The 256kb limit hasn’t gone away; it will just creep up on you when you least expect it. Suppose you switch compression algorithms to buy more time - now every message detected as base64 will have to rotate through a bunch of decompression algorithms to find one that generates reasonable-looking output. There goes your performance improvement.

            Some alternatives:

            1. Spin up an alternative SNS topic for compressed messages, and have different lambdas processing compressed/uncompressed data. Possibly causes a maintenance explosion if you need to have multiple consumers of both topic.
            2. Prefix your payloads with some metadata (at least a version number), in a way that’s fast to parse.
            3. Use a faster language if speed really is that important, and probably pick up safety guarantees too given recent advances in popular typesafe languages.
            4. Store large bodies elsewhere, like S3 or DynamoDB. Adds a lot more complexity to your messaging system.
            1. 8

              Devil’s advocate: why impose a maintenance or complexity burden up front? If you can use a good enough solution right now, you can attack the complexity later on when you have a better idea of how the problem is going to change. If it never comes up then you avoided over designing your system.

              That said, I don’t totally disagree with you.

              1. 10

                Alternative 2 is the best idea imo. Encoding the message “kind” in a header can be very cheap and very fast.

                1. 1

                  Agreed.

              2. 4

                We were already considering option #4, but as already mentioned by other comments, we didn’t want to add that complexity upfront at the moment.

                1. 3

                  FWIW, I worked on a system dealing with SQS’s similar limit, and we’ve also dealt with compression and multiple formats. Observations, though I’m not sure they change anything for anyone:

                  • We could be confident that compression does what we need because we know the kind of content we put in. For us it’s batches of similar items and we get much more than gzip’s typical ~3:1 for text or such.
                  • We temporarily ran a fork of the system where individual items could be huge, and implemented option 4: a JSON struct in SQS with some metadata and a pointer to S3. (We also switched to zstd, which can better handle repetition across large items.) There was enough work to do on each batch that the S3 latency didn’t really matter.
                  • Hacking in a second format without an explicit version number was easier in practice than in theory. We run the producer and the consumer. Our compressed messages predictably started with gzip’s magic bytes base64’d, and our JSON pointing to S3 with {. A msg.startswith is negligible even compared to just the SQS fetch, and we still have a lot of magic-number (or other format-distinguishing) space open.
                  • SQS and SNS also both have message attributes you could use for version numbers, S3 pointers, etc. Seems like you’re essentially having your AWS library do a serialize/deserialize for you, with no advantage in terms of size limit or billing, but they still may be convenient.

                  Enough of the code was the application (as opposed to the fetching/decoding) that I’m not 100% sure if we used msg.startswith or try/except and the difference is small compared to other work being done and other code.

                1. 1

                  Another place you get a a 1/(1-x) term like that is modeling garbage collection: the x is the fraction of available memory you’ve filled with live data (i.e. non-garbage). In a simple model with full GCs only, if you’ve got 8 GB to play with and 4 GB of live data, you can allocate 4 more GB between each GC. If you have 7.9 GB used, you can only allocate 100MB, a 40x increase in work due to a <2x increase in live data, thanks to the magic of 1/(1-x). As you approach live data in all available RAM, you hit the wall. (Java can raise OutOfMemoryError because of a situation like this, where it can technically allocate but has to collect way too often.)

                  There’s a big gap between the toy model and a useful model–short-lived allocations are cheaper in generational GCs, you care about how costly and background-able the GCs themselves are, different runtimes pace GC differently (Go doesn’t look at physical memory size, Java does), etc..

                  Still, when I think about how, for instance, the “peak JavaScript” argument (that mobile Web apps would never speed up much after 2013) didn’t pan out, there’s obviously a lot of credit due to better JS engines and faster and more-core’d CPUs (among other things!), but it’s also pretty important that today’s phones pack a few times the RAM of 2013’s–it moves the “wall” on that 1/(1-x) graph a ways to the right.

                  1. 3

                    The thing about a standard (or mess of standards) is it’s hard for a blog post to change it, but a thing we can do is be specific documenting text import/export in our own apps. Like:

                    • Are headers required or optional or banned?
                    • What encoding and newline sequence are used?
                    • Will you take compressed files?
                    • How does quoting/escaping work? What characters must be quoted/escaped? Are other escapes accepted (exporter might want to use them because other tools want them)?
                    • If there are nulls, how are they represented? Are they indistinguishable from some string value?
                    • What format do dates (or other complex data) use?
                    • What happens on bad input: stop processing mid-file, skip the line, warn and try to do your best, or what?

                    And similar questions for exporting text. You can also answer by reference, e.g. “we’re using [Python’s CSV module/Apache OpenCSV/…] with default settings” or “we’re aiming for compatibility with [Excel/this MySQL LOAD DATA INFILE statement/this Postgres COPY statement]”. (Maybe a mix of the two.)

                    FWIW, my work app imports CSVs and documents maybe half of the things in that list right now, so not speaking from the high ground, plus some of our answers are “we run heuristics on the file to try to guess the right thing”, which is fun. Sadly I probably know more of the pitfalls from using other’s software than I know from what I work on.

                    Besides Parquet/ORC/Avro, I kind of wish newline-separated JSON were used as a data interchange format a bit more. You can take advantage of languages’ existing support, with gzip size can be reasonable, and support for large collections is a bit better than sending one big JSON object.

                    1. 3

                      My cognitive dissonance is off the charts these days with Microsoft. Are they the goodies?

                      1. 24

                        They are a big company doing many different things. It saves a lot of heartache to ditch the goodie/baddie scale and consider each activity on its merits. And in this case I think they’re onto something good. I for one would sacrifice 10-50% JS speed to rule out an entire class of common security bugs.

                        1. 2

                          ditch the goodie/baddie scale and consider each activity on its merits.

                          I gradually changed to this thinking approach after I crossed 33-34, however I struggle to communicate this thinking approach to people. Is there a name for this mental model?

                          1. 3

                            People are complex, organizations change over time. Initial impressions matter a large amount, but people who are interested update their opinions on the basis of new information, while retaining history.

                            One way to do this is bayesian estimation. The two primary problems with bayesian estimation are that people are no good at keeping track of their priors and are no good at estimating new probabilities.

                            It is reasonable to assume that Microsoft, at any time, is doing something clever, something stupid, and a lot of things that they think will make them money in a medium-term future.

                            1. 1

                              Right tool for the job. It almost always redirects the conversation to the technical merits than moral policing. That does not mean moral and ethical considerations are useless, it just helps better calibrate the discussion.

                        2. 17

                          They are:

                          • Giving schools free copies of their office package in order to maintain their OS and productivity software dominance on the desktop.
                          • Lobbying public sector heavily to get locked into Azure.
                          • Using their dominant position to get the whole public sector as well as kids and students onto Teams and O365 with a recurring subscription.
                          • Cross selling One Drive heavily on their non-enterprise Windows editions.
                          • Showing ads and install bloatware in their non-enterprise Windows editions.

                          They want nothing else than total platform dominance and they don’t care about the little people, obviously. The question is, do you consider total dependence of our administrations and of the less-well-off on them a goodie?

                          1. 4
                            • Microsoft fights to get big Pentagon contracts. GitHub can’t even drop a small ICE contract, because Microsoft doesn’t want to offend the military-industrial complex.
                            1. 3

                              Thank you.

                              1. 6

                                No problem.

                                They also care about developers and fund some important research, so it’s not all black & white. Just to be fair.

                              2. 1

                                Google, Amazon, Apple, etc do things too, it’s not like MS is alone in their, MY PLATFORM ONLY perspective.

                                1. 1

                                  True, but I have yet to see GAA employees to write local laws.

                                  I am pretty sure their lobbyist are real busy in Brussels and Washington, though.

                              3. 8

                                I think, very approximately, corporations which sell platforms tend to act like arseholes when they have the upper hand (in terms of network effects and lock-in) and saintly when they don’t.

                                e.g. 10-14 years ago, with win32 being the juggernaut of “we already sunk cost into desktop apps on this platform” and the iPhone only just being released, Microsoft were barely even bothering to disguise their efforts to extinguish open protocols. Meanwhile Apple were pushing html5 hard as the big thing that would allow software to work beautifully on all platforms.

                                Whereas now we have Microsoft very much less dominant and Apple have a near monopoly on phones purchased by the subset of users who spend money on their phones. So Microsoft are promoting open everything, and Apple are doing things like strategically dragging their feet on html5 features, adding horrendous bugs to iOS Safari and not fixing them for months to years.

                                1. 5

                                  Nope. I think they’re doing a great job of “embrace” though.

                                  http://www.catb.org/~esr/halloween/

                                  1. 5

                                    No such thing–these companies look out for themselves, period. But you are better off with more than one company duking it out, because to compete they will (sometimes) do things that are pro-consumer or pro-developer or whatever.

                                    Microsoft on top of the world tried to keep the Web stagnant once they’d killed Netscape, and undermined the growth of Linux and GPL software every way they could. Microsoft as a bit more of an underdog likes Web standards, runs GitHub, and builds the favorite open-source dev environment of a bunch of new coders.

                                    Same with, for example, AMD and Intel. AMD got competitive and prices of high-core-count chips plummeted–great! Now with a chip shortage and a solid follow-up product, AMD is in a position to start raising prices, and they are. Intel getting its manufacturing improvements on track would probably give us more of a price war in certain segments, not less!

                                    I’m old enough to remember when AWS was exciting to smaller devs because you didn’t have to shell out upfront for servers or provision far in advance for spiky loads. 🤣 Now it’s a huge profit center for Amazon and there are plenty of justifiable complaints about pricing (notably for egress) and undermining open-source-centric companies by using their market position to compete with the product developer’s own managed offerings.

                                    Tactically, I guess I want underdogs to reach competitiveness, and for companies to see some benefit to doing good things (e.g. open sourcing things) and cost to bad ones (e.g. pushing out little companies) even when the immediate economic incentives point the wrong way. But in the long run all these companies respond to their economic situation and none of them are your friend.

                                    1. 4

                                      It doesn’t really make much sense to blame the moral character of individual actors when there’s a system of incentives and punishments driving things. If a big company were all saints and angels, they’d be out competed by another company that was willing to play the cutthroat game to maximum advantage. (And sometimes it is worth noting playing the cutthroat game to maximum advantage means doing good things, and sometimes good things come out of bad actions too, but if the market+political winds shift, they’ll (eventually) shift with it or go out of business.)

                                      They’re doing what they think will make them money without getting another nasty visit from the government regulators. So is Google and Apple and Mozilla. None are good or bad per se, they’re all just navigating the same sea.

                                      1. 3

                                        I’ll add to mordae that they were using legal muscle to extract billions in patent royalties from Android. I saw one article claim that stopped. I haven’t followed it in a while. You could check to see if they still patent troll others in general. I mean, that’s clearly evil.

                                        On showing ads, the traditional logic for desktop software, games, etc is that people who pay don’t see ads. Microsoft started putting ads in paid products, including the Xbox. After gamers protested, they expanded how many were on the screen instead of reduced it.

                                        One might also count the integration between single-player games with Xbox Live. I shouldn’t have to login to an online service to play an offline game. If I am offline, I shouldn’t lose access to my progress, items, etc. I recall having to deal with stuff like that on Xbox. The nature of online service is they eventually discontinue it for some products and services. Then, they’ll stop working at all or not work as well. All of this is done to maximize sales of new products using Microsoft’s and their suppliers’ platforms.

                                      1. 7

                                        I like the article, but it seems mired in yesteryear. Today we’ve got Percona XtraDB Cluster, Timescaledb, Citus for distributed PostgreSQL, CockroachDB – just to name a few.

                                        I’d be interested in hearing the war stories of folks running any of today’s cluster extensions for MySQL/PostgreSQL/etc.

                                        1. 24

                                          Percona Cluster and Citus require substantial operational expertise and hand-holding from the application code to work properly. Vitess is probably the least tricky of the database sharding layers, and even it requires lots of in-depth knowledge. You can’t just put these databases in Kubernetes / AWS Auto Scaling Groups / whatever and let ’em rip. Even NoSQL databases like Cassandra—designed from the ground up to be active-active—require dedicated provisioning and special care by ops teams.

                                          As a database person, I think the author is basically right. But I would instead say that good ideas have to face reality at some point. Reality usually includes concepts like “users don’t like data loss” and “distributed consensus is hard.”

                                          1. 4

                                            could someone take a moment to elaborate briefly on what kind of experience and special care these sharded sql systems need? i have seen this work happen from a distance don’t quite know what the pitfalls and pain points are. how much of this is avoided when using “sharded database in a box” services, and how much can’t be insulated from the application layer?

                                            1. 3

                                              I’m also curious about real-world stories. I looked around a bit about Vitess. Slack and Square Cash shared info about their migrations. Alas, incentives are probably against sharing all the bad parts, but e.g. Square talked about deadlocks in their migration.

                                              There are nearly-inherent tradeoffs to sharding. Your choice of shard key for each table (user ID, product ID, etc.) has performance implications; queries spanning shards or using cross-shard indexing will be slower (more hops), and cross-shard transactions are much slower if supported, because of the coordination. You tend to need management infrastructure, e.g. metadata servers and something like etcd or zookeeper. Having more stateful boxes means you need smooth automated spin-up, monitoring, backups, and so on.

                                              Seems like there’s work/complexity/resource overhead even for well-packaged sharding, but 🤷‍♀️. When you need it, you need it, and running a larger monolithic database isn’t worry free either. The growth of mature, relational distributed database tools (and some experience at the edge of when a single DB makes sense) has moved me a bit from “stay monolithic until you can’t do otherwise” (that’s a long time!) to thinking you might look at options when you’re feeling real growing pains even if you could stretch it out longer.

                                              1. 10

                                                I work at Cash and have built multiple applications on top of Vitess. Everything you said matches my experience.

                                                • Sharding is hard. Query pattern lock-in.
                                                • Sharding is hard. Cross-shard anything is strongly NOT recommended.
                                                • Sharding is hard. The application will have a lot of special code for it.

                                                For Vitess in particular:

                                                • Running Vitess and MySQL more than doubles operational complexity.
                                                • Vitess is not popular. Good luck 🤞🏾
                                                • I found the admin tooling for Vitess very poor.

                                                We’ve deprecated Vitess for all new services. Most teams have moved off it … though a couple critical services linger. We’ve got dedicated teams working on them.

                                                We’re increasingly an Aurora and DynamoDB shop now.

                                                1. 5

                                                  Find an alternative to DynamoDB before you outgrow it. Once you hit that wall it is insanely expensive in engineering to work around it.

                                                  I helped run one of the largest DynamoDB tables in all of AWS and we ran into such fun problems as “our backup restore was cancelled because they didn’t have cap” and “autoscaling is failing because it’s taking so long to scale it’s timing out.”

                                                  The cloud is fun until it runs out.

                                                  We just use S3 today it works much better.

                                                  1. 2

                                                    How do you use S3 as a replacement for DynamoDB?

                                                    1. 1

                                                      You treat the entire thing as a KV store. Your bucket key path is whatever you were using for dynamo. Cache the heck out of data in a redis cluster and you should be good to go.

                                                      I know this sounds simple and it is because I don’t have all the details, another team did it. It took them a few months but the outcome was really good.

                                                    2. 1

                                                      To share a bit of “hidden knowledge”, this is true for everything AWS (and I suspect most cloud providers) has, not just DynamoDB. We’re one of AWS’s largest customers at $WORK and we frequently run into all sorts of scaling issues with their services. But we also do ensure that data that sits in DynamoDB is small and doesn’t run into hot shard issues. Larger data usually sits in dbs that we manage which are plagued by all the problems everyone in this thread is discussing.

                                                    3. 1

                                                      How’s Aurora working out for you? We’re considering using the PostgreSQL version.

                                                      1. 2

                                                        We use the MySQL version and not the serverless offering. It’s given me no new pain when contrasted against MySQL instances hosted in our data centres. It’s mostly taken away vertical scaling pain.

                                                        I’m waiting for the other shoe to drop.

                                                      2. 1

                                                        Ah, wow, that changed my outlook on Vitess substantially, even having known parts of it in theory. I wish info like this–not the launch post type stuff but what people learn from day-to-day use–was public more often, though I get the reasons it’s hard for that to happen.

                                                  2. 1

                                                    One of my clients uses a relatively small PXC (Percona XtraDb Cluster, for those unaware of the “official” name) setup, with three nodes.

                                                    It has some operational overhead in terms of the service itself obviously, but I’m struggling to think many instances where application code has to allow for the clustered nature (or side effects thereof) specifically.

                                                    Could you elaborate on what you meant here?

                                                  3. 2

                                                    You did not mention but TiDB gets my vote.

                                                    1. 2

                                                      We’re experimenting with TiDB. What’s your experience?

                                                      1. 1

                                                        Overall positive. Easy to use for sure. I need more experience with stability and reliability.

                                                  1. 3

                                                    Vitess looks really interesting (but see below) for scaling out a lot of projects targeting MySQL. Notably, it’s mostly MySQL-compatible, so you can just run some framework targeting MySQL on it.

                                                    (EDIT: see quad’s comment, though. Significantly less sunny picture than I paint below and they’ve actually used it!)

                                                    It supports sharded keyspaces, and can sometimes execute queries/joins within one shard. It can make trickier queries Just Work with scatter/gather and other execution strategies. It has some operational tools for moving tables, doing backups, etc. It has some large users and there seems to be a developed operational approach around it (e.g. shards of 256GB instead of as big as the server can support). There are some neat primitives like VReplication that I suspect you could do useful stuff with.

                                                    There can be blockers–it has its own SQL parser so it does not support quite 100% of what MySQL does (there are examples in docs and the test suite). The cost/benefit depends a lot on whether each of those features gives you something new, replaces something you already have in another form, or simply isn’t useful. [And adding in the real-world experience quad mentioned, I’m a lot more hesitant than before! Interesting tech, at least.]

                                                    1. 3

                                                      There are a couple big things that keep query languages legitimately different from general-purpose PLs, I think. One, query planners help us out a ton, so you want the language to be transparent to the planner so you can declaratively say what you want simply enough that it can figure out how to get it. You need to be careful inserting wasm blobs or such to not break that. Two, if you move a lot of your app’s computation into the DB, you might pay (already true with piles of stored procs today)–wonky code can overload or crash your DB not just the application instance, and in general you might have to scale your DB deployment for computation needs more which can be suboptimal (especially when DBs were harder to scale period!).

                                                      None of that justifies all the quirks of SQL; it wasn’t handed down from on high. More types, more conciseness, and making more imperative-style control a reasonable option all seem interesting. Weird to consider why SQL hasn’t gone anywhere, and even seems to be having a second heyday after NoSQL was the hot thing for a stretch.

                                                      Over the last decade or two we’ve had many general-purpose programming languages get to the point where they’re entirely practical to build big new apps in when, in ancient history when I was learning this stuff, it seemed like the implementations and ecosystems were so big and difficult to create we would kind of be stuck with a handful forever. Alternative languages existed, of course; the “practical to work in” part was the hard one. Many things changed, hard to pin down their relative importance–pieces of compilers/runtimes being reusable (LLVM, JVM/CLR, JS backend, etc.), collective learning about how to nurture a language ecosystem (importance of docs/introductions, stdlib, good FFI to fill gaps early on, easy ways to share libs, etc.), better hardware lowering the performance bar for “usable”, more programmers becoming polyglots, just a bigger population of programmers in general to write/use all this stuff?

                                                      So why hasn’t there been a similar explosion in query languages? Maybe we’re missing relational backends (with secondary indexes, query planners, etc.) easily separable from the query language? Maybe the risk aversion is greater? (That feels plausible intuitively, but you were also hosed if you wrote your app in a flaky language and we got over that.) Maybe the effort in DBs has gone to building decent distributed database backends, and we’re late to taking a good look at the layer above that?

                                                      FWIW, I think there’s another, loosely related, oddity of how we do data now: DBs bundle together lots of functionality that may not have to be bundled. Like, if you have two files that don’t fit in memory, doing a sort join doesn’t inherently require offloading the task to another binary that also has a serialization format, a transaction layer, a query planner, etc.; it just needs an external sort function. I think we’re chipping at it from a few directions–SQLite doesn’t have the server, Trino/Presto/Athena query data in its native format, data science libs do interesting things in an imperative way, there’re plenty of KV libraries. These things are each kind of islands now, and it’s interesting to imagine something more like a proper database that is separable into all these pieces (which, yes, is super hard, but we’re dreamin’ here).

                                                      1. 50

                                                        This seems to contradict some statements from the article (assuming it is true) in that it reproduces non-trivial GPL code and relicenses it on the fly.

                                                        1. 17

                                                          I think what Julia is missing is that this is something that the big coprs can do, but individual developers will get sued into the ground.

                                                          1. 7

                                                            “Copilot relicences the code on the fly” is a bit of an iffy characterisation. Armin gets the inverse square root, then goes back to the top of the file and then says “fill out this thing for a license!”. This isn’t some fully-automated process, this is purposefully being guided.

                                                            I don’t think that Armin was implying that this tool should have properly guessed the license here. There’s already enough stuff to be said about Copilot, there’s not that much of a need for silly demos like this to be taken as a legal argument.

                                                            1. 7

                                                              The licensing portion of that demo is basically irrelevant. If Copilot lifts a function verbatim from one codebase and puts it into another, then the licenses of those two codebases are what matter.

                                                            2. 5

                                                              Corollary: even if you put in some sort of wrapper/check to avoid outputting GPL-licensed snippets like this, or try to add attributions or whatever, the network contains a copy of the GPL code (encoded as weights and so on) so you have to deal with whether that copying/derivation is legitimate.

                                                              It doesn’t necessarily make it go away if the network sometimes produces output way different from any input (my copies are still copies even if I also do original work), or if copies are often inexact (I can’t get away with distributing that photo editor just ‘cause I edited its splash screen to say “Photoslop”). Law is currently that humans can add a lot but still infringe (see the “My Sweet Lord” lawsuit), weird to give a well-funded company’s computer program a pass. In general I think IP laws should be weaker, but I don’t think they should be selectively weaker for companies skirting open-source licenses!

                                                              Seems like the robust solution is using code you actually have permission to use, either because the licenses are explicitly compatible with hoovering up source into a commercial product, or copyright owners relicensed the code for this use, e.g. if MS used their own code or got separate permission from companies or open source authors to train on their stuff. (Some risk of your model getting ‘infected’ by someone else’s bad license/authorship labeling though.) Or (I think?) you could even have a GPL-licensed model that produces GPL-licensed code from GPL-licensed source.

                                                            1. 13

                                                              It’s interesting the SMART Health Card standard implemented here is entirely incompatible with the Digital COVID Certificate standard (Interoperable 2D Code, pdf) being rolled out in the EU (and currently used for the digital NHS England COVID Pass).

                                                              Perhaps the IATA Travel Pass will be more successful as a unifying standard.

                                                              1. 5

                                                                Just for fun, the contents of the EU covid cert have a much more concise-looking schema than the US one (less XML-y deep structure and magic URLs). And the European container seems to be CBOR + Base45 vs. the US one JSON base64’d then run through a transform that doubles byte count turning everything into decimal digits. Both use gzip. (Ed: turns out QR codes have a numeric encoding that makes three decimal digits only take ten bits, so the US way is transmitting 6 bits in 6 and 2/3 bits on average, ~90% efficient. And Base45 gets 16 bits in three 5.5-bit chars, ~97% efficient. Now it all makes more sense!)

                                                                Interesting that both versions seem to fit in that size QR code (must just be able to hold a lot); I’d’ve thought even with gzip, everything in the US structure would be a tight fit.

                                                                1. 3

                                                                  Note that what the US one is using is a standardised interoperable healthcare format called FHIR. The json representation looks pretty verbose, but handles many things you’d forget when coming up with your own format to represent healthcare data.

                                                                  Just look at the FHIR R4 definition for HumanName in context of Patient

                                                                  • name HumanName 0..*: A person may have 0, 1, or more names
                                                                  • For each HumanName:
                                                                    • use {usual, temporary, official, nickname, maiden, ...}: The context of this HumanName; does this person use it as a nickname, is it the person’s maiden name, …
                                                                    • family string 0..1: May or may not have a family name
                                                                    • given string 0..*: 0 or more given names (usually surname)
                                                                    • period: 0..1 Period: The time period this name was/is/will be used

                                                                  And this is just a small extract from just the HumanName data type. FHIR also has a system to manage logical IDs as well as external IDs (i.e. if a Patient is tracked in different databases in a hospital), support for various code systems used in healthcare (ICD-10, CPT, …), the most complex/complete system to handle temporal information I’ve seen, a super-integrated extension mechanism, …

                                                                  The whole documentation, data schema definition and basically everything is also completely machine-readable.

                                                                  It’s very complex, but I recommend everyone who does some sort of data modelling to take a look at some of the concepts. It’s a great inspiration.

                                                                  Source: I’ve been working with FHIR for a few years now :-)

                                                              1. 2

                                                                No deep insights, just my WSL-centric experience, but I definitely felt “at home” sooner in the last switch from Ubuntu to Windows than when I used something other than Linux in years past. Cross-OS VSCode remote sessions work well: language servers, etc. can run remotely or in WSL. There’s a reasonable first-party terminal (the one you install, not built in) and build of OpenSSH. Haven’t learned PowerShell but I sometimes use the tools I already knew from the Linux side to find things, munge files, etc. in Windows. The shell and other built-in stuff are keyboard-friendlier than before (e.g. Super and type to search, switch apps with Super + number, Alt-Tab and arrows). Gotta look at the new PowerToys the post mentions.

                                                                I’ve also got complaints about Windows, but this isn’t a review. Just neat how much the default OS shipped with a bazillion computers has changed from the perspective of a dev with mostly Linux experience.

                                                                1. 2

                                                                  I’ve also got complaints about Windows, but this isn’t a review. Just neat how much the default OS shipped with a bazillion computers has changed from the perspective of a dev with mostly Linux experience.

                                                                  WOW so much this!

                                                                  When I wrote the article, I didn’t mean to declare a deep, abiding love for Windows. It’s more like “Hey wow in the past when I tried it it was PAINFUL, and now it’s smooth like buttah!” :)

                                                                  I will always love Linux best in my heart of hearts. The idea of a fully FLOSS work environment is super shiny. Maybe when I retire I will have time to dive deep and fix these deal breaker accessibility bugs that make desktop Linux super painful for me to use today, but until then, I’m super psyched to have found an environment that lets me Get Things Done with a modicum of pain.

                                                                1. 110

                                                                  Independently of what burntsushi decides, I think that the following concrete actions would be in order:

                                                                  1. Remove the last half-sentence of the banner that explains how to delete your account. Having it as the last part of the message adds a lot of weight ot it: after you read the banner it’s the main thing you remember, the oddly detailed explanation of how to erase yourself.

                                                                  2. Reformulate the rest of the banner to indicate that this is a heuristic that may be wrong. It’s important when providing hints from data to tell people explicitly that we know the hints may be wrong. Otherwise it feels very different.

                                                                  Let’s rephrase this banner with the explicit goal of doing no harm (better, doing good) when it is wrongly displayed, instead of thinking about the formulation when its assessment is correct.

                                                                  1. 59

                                                                    Speaking as somebody who’s gotten that banner basically permanently on their account, I think a lot of folks here may be missing important context:

                                                                    • The call to delete your account makes a lot of sense if you have to handle users dramatically declaring, in DMs/IRC/elsewhere “One more and [clutches pearls] I shall leave, forthwith!”. This is tiresome from a moderation standpoint and annoying for a community, and if one can nudge somebody into removing their account–in the common case!–rather than continue to engage in a community they don’t fit into (for whatever reason), it’s a win for both parties.
                                                                    • This isn’t some randy sketchy heuristic–the warning links directly to a histogram that shows where you are in the flaggings. It’s deterministic, it’s impartial, and if you’re getting flagged a lot there’s something going on with either your posting or the community.
                                                                    • Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.
                                                                    • One of the biggest reasons for Lobsters coming from the orange site was @jcs being annoyed at opaque moderation. While I might disagree certain particular actions, there is a public log. The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.
                                                                    • There’s a trend for folks to go on IRC, Twitter, or wherever and talk trash about our community, about how @pushcx is powertripping, or whatever. This behavior probably plays well with the circles they’re familiar with and probably feels like punching up, but it’s gotta suck for the person who manages the community to have people he’s tried to work with and accomodate throw it back in his face.
                                                                    • People are really flag-happy, and this is what happens when you are.

                                                                    (Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

                                                                    1. 35

                                                                      Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.

                                                                      This bit deserves signal boosting IMO.

                                                                      Lobsters doesn’t cost anything to join or participate in. The people who run it are clearly doing it for love, not money.

                                                                      Speaking for myself I’d much rather get past the momentary ouch of having a red warning message attached to my account than have the owners or moderators rage quit as a result of feeling UTTERLY drained by the experience.

                                                                      I’m watching the life get sucked out of some very stalwart well meaning people in another community I care about simply due to the sheer VOLUME of constant negative feedback, so I think we all owe it to the mods to suck it up and cut him/them some serious amounts of slack.

                                                                      1. 22

                                                                        <3 Thank you very much.

                                                                      2. 24

                                                                        (Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

                                                                        I would love to see this come back in some form. As someone who joined after that was removed, I feel there isn’t enough information about flags. Every single time I’ve ever flagged something, I’ve been really hesitant because I wasn’t able to find any guidance on what qualifies to be flagged.

                                                                        1. 3

                                                                          I agree. I actually don’t think this is so minor on its own though I can see why you phrased it that way in its original context.

                                                                        2. 23

                                                                          I realize I’m not as active as I might be - I’m bipolar, my ability to spend time on things comes and goes - but I promise that I do still pay active attention to the site and step in when it seems warranted. Also, the mod team has lots of discussions behind the scenes. Just because @pushcx is the face of a decision doesn’t mean the rest of us aren’t involved.

                                                                          I felt I should address that, since you mentioned it.

                                                                          Edit to add: Thank you for the kind words, overall. It’s helpful to know that you and others do see the point of the banner.

                                                                          1. 22

                                                                            People are really flag-happy, and this is what happens when you are.

                                                                            Speaking from experience as someone who’s moderated some decent-sized subreddits – where reddit has both the ability to downvote, and a separate ability to “report” for posts/comments that break subreddit or site-wide rules – this is absolutely the case. The number of people who just spam-click the “report” button thinking it’s some sort of super-downvote that will remove the thing they dislike is astounding. Same with people who periodically decide they just don’t like some type of content even if the subreddit generally allows it, and go mass-report a hundred posts to express their displeasure.

                                                                            1. 11

                                                                              Yeah, this is what I think is fundamentally the issue. A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button. Perhaps in their mind, if the post was deleted, then the opinion no longer exists or is somehow invalid???

                                                                              I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

                                                                              People simply think people who disagree with them are either stupid, or trolling (deliberately saying stupid things hoping for a reaction) or both.

                                                                              1. 12

                                                                                A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button.

                                                                                It’s also true that people sometimes repeatedly post tired nonsense, and that merely (and at obviously low cost) repeating something that’s poorly researched, or broadly offensive, doesn’t entitle the poster to a vigorous debate. Sometimes the use of a flag is the only viable option.

                                                                                1. 4

                                                                                  If somebody posted some tired nonsense then it should be very easy to dispute.

                                                                                  doesn’t entitle the poster to a vigorous debate

                                                                                  You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

                                                                                  Sometimes the use of a flag is the only viable option.

                                                                                  Anybody could deem all your posts ‘tired nonsense’ and just report all of them? That is as legitimate as when you do it. Do you somehow think that your idea of what is tired nonsense is universal and therefore a good metric for when a post should be reported?

                                                                                  1. 8

                                                                                    If somebody posted some tired nonsense then it should be very easy to dispute.

                                                                                    In the case of trolling, the whole point is to trick people into taking the bait, which I believe moves the subthread higher up in the scoring ranks.

                                                                                    You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

                                                                                    I think that detecting trolling is a skill that’s pretty key to the survival of a community, but a good troll will make it quite difficult to resist engaging. Trolling isn’t commonly overt, over the top rudeness. Way more often it is an attempt at looping other community members into a “debate” that ends up in people pointing edgy and charged hot takes at each other.

                                                                                    In its worst form, it will promote socially harmful conclusions through pseudo-from-first-principles lines of reasoning that are aesthetically attractive to the type of people that visit tech forums. To dispute requires dissecting phony arguments, which at a glance, appears to legitimize the “debate” and grant the troll a certain level of community approval, especially if they’re good at giving off the appearance of rationalism. IMO ignoring and flagging this type of content is nowhere near what’s required fully address it.

                                                                                    1. 3

                                                                                      How are you not using an accusation of trolling as a free ‘You lose’?

                                                                                      How do you know that something is true? It is via putting your ideas up for others to engage and debate. But you want to simply circumvent that by calling opinions you don’t like trolls, and then saying that those opinions don’t even need to be debated or engaged because that will legitimise it.

                                                                                      Sooner or later instead of having attacking each other’s ideas so we can improve then, all we will end up doing is claiming the other side is ‘trolling’ so they are immediately wrong and don’t even need to be disproven.

                                                                                      Why even bother defending your ideology against opposition, when you can simply claim the opposition is a foreign spy or a mentally ill person and then getting rid of them?

                                                                                2. 5

                                                                                  I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

                                                                                  Or maybe you’ve gotten more and more trollish over time.

                                                                                  1. 5

                                                                                    Or maybe we have too many politically driven ideologues who are not interested in communication and they simply throw the word troll around to avoid having to engage with their opposition on the idea plane.

                                                                                    1. 9

                                                                                      Nah, I’ve been around just about as long as you have, I can count on one hand the number of times I’ve not seen you be a troll in a thread. I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread. Pretty much all you do is get into fights around here.

                                                                                      1. 3

                                                                                        Devil’s Advocate: Perhaps @LibertarianLlama is among a new breed of Lobsters who are also concerned with the human side of things, not just the technical one.

                                                                                        1. 2

                                                                                          I’ve not seen you be a troll in a thread

                                                                                          I suppose you are one of those people who have a very low margin for what constitutes ‘trolling’.

                                                                                          I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread.

                                                                                          I didn’t realise technical expertise in a subject is a requirement of using this website. Regardless you can just assume that I have no expertise in any subject.

                                                                                  2. 6

                                                                                    I wonder if a literal “super downvote” button would work as a psychological trick. It would act as a normal downvote, but just look like a bigger, angrier button that would absorb more anger from whoever clicks it. (At the same time, possibly rate-limit the actual report button per user…)

                                                                                    1. 8

                                                                                      That’s a pretty cute idea. I’m tempted to say that, for people who have the self awareness to realize that what they mostly need is catharsis, I recommend getting it from their favorite action videogame, or a trip to the gym or whatever… I don’t want to dismiss your idea though, it’s true that it could help to have some sort of reminder or prompt to seek catharsis.

                                                                                      1. 3

                                                                                        Edit: someone has looked at the system and much of this is actually already automated(!): https://lobste.rs/s/zp4ofg/lobster_burntsushi_has_left_site#c_qiipbc

                                                                                        Previous:


                                                                                        OK, another cute - or hopefully even better - useful idea:

                                                                                        I think HN has a pr account setting that ignores flag from that user, I’m fairly certain dang has mentioned at some time that one should be careful with flags. (Now that I think about it it might have been the vouch option he was writing about.)

                                                                                        I’m not really sure how it would work with perfectly transparent moderation:

                                                                                        • would the user get a warning?
                                                                                        • or would it just show up in the moderation logs?
                                                                                        • maybe it is a fraction so if a user has 1/5 flag weight it takes more than 5 such users to flag before it counts as one ordinary user? Then the moderation log could still say “flagged based on user feedback” or whatever it say but omit the details about there being 10 flag abusers counting as two full votes and 3 ordinary users?
                                                                                        1. 11

                                                                                          Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

                                                                                          Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

                                                                                          1. 1

                                                                                            Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

                                                                                            Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

                                                                                      2. 5

                                                                                        The moderators of r/TheMotte (a subreddit which has the best moderation IMO) observed pretty much the same:

                                                                                        No one who has seen a mod queue would be surprised by this. Even those of you who are always very calm and nonconfrontational, with middle-of-the-road opinions, might be shocked how often your posts are reported. Some of the most reasonable, unspicy posts get marked as “It’s targeted harassment.” Some people will report “Inflammatory claim without evidence” just because they don’t agree with the post.

                                                                                        I pretty much started brushing aside (like several other users reportedly do) the red warning that lobste.rs throws in your profile for this reason, as well as for that condescending final phrase “Reconsider your behaviour or take a break” which rests on the erroneous assumption of there incontrovertibly being sound rationality in this community’s flagging behaviour.

                                                                                        I have a theory that this became worse after the site removed the downvote functionality thus removing a potential anger-outlet for users, who have now shifted to channel that affective energy via flagging. This theory however can only confirmed by doing a simple analysis of the data (was there an increase in flagging right after disabling downvoting?)

                                                                                      3. 12

                                                                                        It’s irrelevant that it is impartial if it is exploitable. Which it very much is.

                                                                                        So the system allows for a mob to harass an user into leaving the site, but you claim that it is important to note that anyone can be harassed. That is the matter subject to challenge here, not “important context”.

                                                                                        Yes people are trigger happy, for that reason, the flagging feature might be counter productive.

                                                                                        1. 5

                                                                                          you claim that it is important to note that anyone can be harassed.

                                                                                          I’m not sure I see where you got that from my post.

                                                                                          The ability of the mob to harass users here exists, and is exploited, at scale and quite apart from this notice that “yo, you’re getting a significant number of flags.”

                                                                                          It doesn’t make sense to optimize for people who run away when told others disagree with them.

                                                                                          1. 21

                                                                                            No one wants to be told they don’t belong here. I really enjoy Burntsushi and a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

                                                                                            Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

                                                                                            1. 6

                                                                                              OK I see an important distinction here. I tried to point this out to burntsushi on Twitter.

                                                                                              If you read the message he got VERY carefully, it sets forth a very specific sequence of potential and suggested routes to mitigation. They are:

                                                                                              1. Take a breath. Take a break. Step away for a bit and think about whether maybe the problem might exist at least partly in your head and the resultant behavior you’re exhibiting in the community.

                                                                                              2. Talk to a moderator about the situation.

                                                                                              3. Then, and ONLY then, does the message point out that you can if you so choose also delete your account.

                                                                                              Having gotten the big red warning myself a ways back, I DO sympathize that it’s an ouchy thing to have happen in the moment, but I strenuously believe that the intent here isn’t to make anyone feel unwelcome. Just the opposite, my read of the intent is an earnest attempt on the part of the community to help people moderate their behavior, and if that’s not possible or applicable, to seek other options for mitigation.

                                                                                              1. 11

                                                                                                No one wants to be told they don’t belong here.

                                                                                                I’d agree, but we have done very poorly by some of our users in that regard already and the mob likes it that way. Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff? Peppridge farm remembers.

                                                                                                There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe–and they’ll get even angrier if you suggest that maybe, possibly, we should focus on something else.

                                                                                                So, while I agree in principle with you, the community has through its actions demonstrated that it does not care.

                                                                                                Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

                                                                                                I don’t believe that it is. You are, of course, welcome to your own interpretation.

                                                                                                1. 14

                                                                                                  Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff?

                                                                                                  IIRC he had a pattern of problematic behavior, and also I’m not sure how much of an expert in ESE he actually is: his book is overwhelming, but keeps giving me bad vibes. I’m struggling to explain what it actually is, but I’m really suspicious that there are serious problems with it.

                                                                                                  1. 1

                                                                                                    I’d definitely be interested to hear more.

                                                                                                    I’m struggling to explain what it actually is

                                                                                                    Mind giving it a try? I couldn’t see anything particularly bad (relatively speaking) in their comment history, nor anything related to the ban rationale. Seems like you’ve been paying more attention, haha

                                                                                                  2. 14

                                                                                                    There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe

                                                                                                    The basic problem with statements like this is that nearly everyone can feel it applies to the angry mob that’s disagreed with them.

                                                                                                    Personally I’m on record as believing that there is no way to separate “politics” from what we do (simple example 1, simple example 2) or to write software without considering the impact it will have on people.

                                                                                                    But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue. The hypothetical guy (example 2 above) with no criminal record who can’t get bail because the system could only store his name as a bunch of “aliases” is “politics” to me, but apparently not to many other people (it’s just “falsehoods programmers believe about names”), while holding a software project’s leader accountable for misbehavior is “politics” to many of those people and common sense to me. A staggering percentage of actual “political” fights really are that sort of thing.

                                                                                                    1. 3

                                                                                                      But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue.

                                                                                                      I mean the proof is at hand right? A “let’s not discuss politics” non sequitur as successful troll.

                                                                                                      1. 0

                                                                                                        But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted

                                                                                                        I don’t think I’ve ever seen this on Lobsters, only on others sites. Do you have any examples?

                                                                                                      2. 1

                                                                                                        I think we disagree about what it means to be thin skinned. I thought you were admonishing the people who leave into toughening up. If that is the case, I think it will lead to either an echo chamber or a dungeon of trolls or both.

                                                                                                        But based on your other comments, I don’t think you meant what I interpreted.

                                                                                                        You in this context means, us, or the offended user that flags people into quitting. How about we make flags public and you can only spend them in certain ways? Do comment points even mean anything? There is a huge distinction between “this person is harmful” and basically anything else. if someone is harmful, they should be removed. Otherwise … if you don’t want to see their posts, then that would be a personal choice.

                                                                                                        1. 2

                                                                                                          Making flags public is a terrible idea.

                                                                                                          1. 2

                                                                                                            I agree. I DO think that people here are overly trigger happy with the flagging, but given other comments in this thread I’m willing to believe this is simply human nature at plan and unavoidable. Making flags public would be in effect shaming someone for trying to make a small contribution to the community by flagging a post.

                                                                                                            1. 2

                                                                                                              I feel that maybe 60% of flags are not trying to contribute to the community, but rather to avenge themselves, or to have an outlet for disagreement without having to engage. It’s part of the problem with the Internet at large: for digital natives, disengagement is no longer an option. People can’t just ignore stuff they disagree with; they have to respond to it, and if it’s a troll, they shouldn’t engage directly, so they instead engage indirectly by flagging. There’s a reason why Twitter’s first option when reporting is “I disagree with this”. It’s just human nature.

                                                                                                            2. 1

                                                                                                              Because? Without justification your statement is opinion, what am I to do with that?

                                                                                                              1. 1

                                                                                                                Because it can lead to retaliatory, tit-for-tat flagging of the flagger’s comments. Mods can already see who flags what and can take action against those who misuse it.

                                                                                                        2. 2

                                                                                                          a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

                                                                                                          Who are “people like that”? To me, this sentence seems to extrapolate based on a single data point, which is based on some observations of another person. That is to say, we don’t know Burntsushi well enough, and maybe no matter how we setup the system, he would find something to quit. And striving to optimize for one would put burden and alienate others.

                                                                                                          1. 5

                                                                                                            people like that

                                                                                                            Hardworking, smart, helpful, writes a lot of great Rust. Reddit and github

                                                                                                            I am talking about specifics not a generalized idea of a Lobste.rs user.

                                                                                                          2. 0

                                                                                                            Since that person left, I’m not getting a flag on every single of my comments anymore.

                                                                                                            An interesting coincidence. ¯\_(ツ)_/¯

                                                                                                            1. 0

                                                                                                              Obligatory “correlation is not causation”.

                                                                                                          3. 5

                                                                                                            That is the problem right there. It.s not a “disagree” link, it’s a “flag” link. Obviously, looks like there are people abusing it because they disagree with something rather than unacceptable behaviour being reported.

                                                                                                            Personally, I wouldn’t take it too seriously much less posting on other websites about it. But it is rather shitty to be told to take a chill pill just because one’s opinion is not widely accepted.

                                                                                                            1. 1

                                                                                                              Accounts with low “karma” can’t downvote but can flag. Perhaps this needs to be looked at..

                                                                                                              1. 6

                                                                                                                There are no downvotes, only flags.

                                                                                                                1. 2

                                                                                                                  TIL. I stand corrected

                                                                                                                  1. 1

                                                                                                                    Thanks for clarifying this. I was wondering where the downvote links were.

                                                                                                                    1. 4

                                                                                                                      iirc the downvotes were the flags, just with a downvote arrow instead of being hidden.

                                                                                                              2. 3

                                                                                                                It doesn’t make sense to optimize for people who run away when told others disagree with them.

                                                                                                                I think discourse is improved when all participants feel safe. Then instead of spending energy in handling negativity emotionally, they can make better arguments. Moreover I don’t want to see valid viewpoints shut down due to mob shouting. I think human discourse works well with a variety of opinions.

                                                                                                                1. 16

                                                                                                                  I think human discourse works well with a variety of opinions.

                                                                                                                  I agree!

                                                                                                                  I think discourse is improved when all participants feel safe.

                                                                                                                  The problem we’re seeing in the last several years is that there are people who will never feel safe, at least not without massive sacrifices from everybody else in a community, often including systematic self-censorship and topic avoidance. The great online community question of the age is how to strike a balance between being welcoming and letting your demographics become taken over by–for lack of a better way of putting it–people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem.

                                                                                                                  1. 9

                                                                                                                    people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

                                                                                                                    I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

                                                                                                                    1. 15

                                                                                                                      An (only slightly contrived) example:

                                                                                                                      You run a message board that hosts tech content.

                                                                                                                      On one extreme, you tell everybody “hey, we want you here! everybody is super nice and you’ll never be criticized!”.

                                                                                                                      After some period of time, you end up with a bunch of users of C and a bunch of users of Rust.

                                                                                                                      Some of those users cannot bear any perceived slight of their chosen language. There are C people that go off in a tizzy if anybody points out the memory safety issues in C, and there are Rust users that freak out if anybody complains about long compile times. The house rules–because remember, we’re optimizing for welcoming and that means supporting everybody, no matter how thin-skinned they are!–end up including “Never talk about shortcomings of another language.”

                                                                                                                      Since engineering is all about tradeoffs, and since no language is actually good for all things, strict adherence to this rule ends up with only the most optimistic and vapid discussion–worse, it infects other topics. It’s really hard to explain why you might like immutability for concurrency if you can’t talk about memory safety issues because you’ll trigger the C programmers. It’s really hard to explain why you like dynamic or interpreted languages if you can’t talk about how much long compile times suck because you’ll trigger the Rust programmers.

                                                                                                                      On the other extreme, you tell everyone “hey, we talk about tech and things on their own merits here, and if you can’t stand commentary that offends, fuck off!”.

                                                                                                                      Our C users and Rust users come in.

                                                                                                                      Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

                                                                                                                      Obviously, there will be users who would be knowledgeable assets to this community but who are squicked out by this stuff and leave. There are other users who can put up with the rampant nonsense but who don’t want to spend the cycles required to weed through the bullshit to talk about what they want. Over time, the community stagnates unless it can find a way of attracting and retaining new users, and odds are that’ll be based on its reputation for no-holds-barred “discussion” rather than actual technical insight…this userbase will also skew towards younger folks who have a deficit of experience and all of the time in the world to share it. This is, incidentally, how /g/ works.

                                                                                                                      ~

                                                                                                                      I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

                                                                                                                      My point isn’t to advocate for one or the other, but to point out that it’s a central question of community.

                                                                                                                      It’s also a question whose answer shifts over time, as groups tweak the trajectory of their tribe.

                                                                                                                      1. 8

                                                                                                                        Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

                                                                                                                        And there’s a pretty easy way to fix that – kick ’em out. There are communities which do that.

                                                                                                                        Because as the saying goes, there are two ways to be a “10x programmer” and one of them is to have such an awful effect on people around you that they drop to 0.1x their prior productivity while you stay average. And even if someone really is “10x” (whatever that means) on some aspect of technical skill or knowledge, it’s still likely to be a net loss to put up with “asshole” behavior from them.

                                                                                                                        1. 2

                                                                                                                          This is good advice and knowledge for me as I try to build one or more Open Source communities. I think you are correct that such communities should have no room for such users.

                                                                                                                          1. 1

                                                                                                                            For the purposes of that thought experiment, we take the position in the second case where we do not eject members for being assholes.

                                                                                                                        2. 3

                                                                                                                          I saw another attempt at creating a HN clone go down a few years ago, not because of flaming or lack of interesting users but because some early users decided they and everyone else decided they needed to be much nicer than HN.

                                                                                                                          I quit commenting soon after.

                                                                                                                          You can check my comment history, I’m not at rude person but those rules made me feel unsafe. I started carefully considering each word. Then stopped posting,

                                                                                                                          I wasn’t the only one. The “super nice” crowd didn’t have anything else to contribute and the site was down a few months later.

                                                                                                                          1. 8

                                                                                                                            Honestly, one of my main issues with HN has always been the focus on superficial civility. You can be an absolute horrendous awful drive-others-away-in-droves trainwreck of a poster there so long as you do it the “right” way. And you will be valued and rewarded for it.

                                                                                                                            A real “anti-HN” would care less about those superficial aspects and more about dealing with the actual trolls and assholes.

                                                                                                                        3. 4

                                                                                                                          people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

                                                                                                                          Perhaps I don’t see this as an issue. I believe, in my ideal community (where I’m stressing “ideal” because it’s probably not practically realizable), that any commenter with a good-faith argument should feel empowered to make that comment, even if others in the community viscerally disagree. There are two problems at hand here, one is defining what “good-faith” mean and the other is protecting arguments from being shouted out. When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements. In my mind, the easiest way to come to a consensus on what “good-faith” means is to clearly define it. This is why I’m a fan of Code of Conducts. They lay bare what a community considers “good-faith”, so the minimum bar of participation is met. Ideally the CoC would change infrequently and have community buy-in so that it acts as a minimum bar to entry.

                                                                                                                          Ideally (again the ideal here not the practice, probably), we can create a community where there is no hostility after agreeing to “good-faith” rules. It’s one thing to enter heated discussion, but I’m hopeful that a community can use its own moderation tools to ensure that a heated discussion does not turn hostile.

                                                                                                                          I just don’t see a world where diversity and hostility are equivalent.

                                                                                                                          1. 1

                                                                                                                            When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements.

                                                                                                                            I make good-faith arguments all the time that seem to fall into the class of bad-faith arguments that @bitrot points out, so I get what you’re saying.

                                                                                                                            I just don’t see a world where diversity and hostility are equivalent.

                                                                                                                            In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄


                                                                                                                            At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

                                                                                                                            1. 2

                                                                                                                              In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄

                                                                                                                              Indeed, I am familiar with the Paradox of Tolerance. My hope is that a CoC or otherwise similar guiding document or constitution would provide a floor on tolerance, but everything beyond that would be tolerated. Not only does that set community baseline rules but it keeps rules distinct, clear, and constant. There’s a reason many real, representative governments go through due process to change laws.

                                                                                                                              At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

                                                                                                                              Perhaps the conclusion I should arrive to is that direct democratic (e.g. “upvote” and “downvote”) discussion sites just lead to dynamics that create and sustain echo chambers. I’m not sure, it’s certainly something I’m mulling about myself.

                                                                                                                      2. 2

                                                                                                                        when told others disagree with them.

                                                                                                                        But that’s not what flagging tells you… If you’ve been doing everything alright, it tells you that others who disagree with you have leveraged a system unrelated to disagreement to make you look unreasonable, unkind, and/or trollish.

                                                                                                                      3. 4

                                                                                                                        I’d be interested in learning if mods have any way to detect brigading, possibly such as by looking at correlations in users who consistently flag the same comments.

                                                                                                                        1. 13

                                                                                                                          Yes, we do. If you look at the mod log you will see that people are banned for this practice now and then. Don’t do it.

                                                                                                                          1. 1

                                                                                                                            I’m not sure why you needed to admonish me like that at the end. I wasn’t probing your defenses. I was merely going to suggest this if you didn’t have it.

                                                                                                                            No need to reply, I know this is a tough time for all the staff.

                                                                                                                            1. 7

                                                                                                                              Sorry, I should have been clearer that I wasn’t directing that at you in particular. I know you were trying to help.

                                                                                                                      4. 8

                                                                                                                        Preface: I don’t have his comments handy and am just looking at the system. Despite this I will be embarrassed if it turns out he was actually making an enormous mess 🤣

                                                                                                                        Also want to add whoever worked on the existing system clearly put a lot of thought into it (e.g. factoring in flags on multiple stories, percent-flagged, and limiting to N users all seem wise, and thought went into the message). Probably no fun to see someone armchair quarterbacking with much less time thinking about the problem space than they have. Hope this at least crosses the threshold of possibly useful.

                                                                                                                        The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.

                                                                                                                        It looks like this is it. My guess is if you’re prolific, it’s inevitable you’ll pop over the hard numerical thresholds sometimes and the 10% of comments is the biggest barrier, followed by the top-30 ranking.

                                                                                                                        10% of comments seems not that unlikely if 1) your recent comments are seen by lots of people (upvoted, early, etc.), 2) they address things a lot of people disagree about, and 3) some % of folks who disagree will flag, which seems likely as long as flagging is one of the easier ways to express disagreement (more below). You could do something like subtract the upvotes/N from the flag count (lots of fiddly options), just because there isn’t a number that represents visibility directly.

                                                                                                                        The top-30 ranking has has the same considerations as flag count, and also if discussion is going well enough, the 30th “worst” person might not be that bad. (Somebody’s the least polite person in a group of absolute diplomats.) The histogram/rank data does seem interesting to expose to the more flagged folks. Given all that, I’m not entirely sure what I think about the wording of the message–definitely already phrased to aim for “stop and consider this” not “you’re a bad actor”, but I can also see how humans get defensive.

                                                                                                                        People are really flag-happy, and this is what happens when you are.

                                                                                                                        (Added clarifications here in [brackets] after friendlysock’s comment.)

                                                                                                                        A nice thing about [the idea of downvotes for simple disagreement, as used on some other sites, is that it lets] people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on. I think the dream behind the current system is that not having [no-reason] downvotes leads to a very pluralist site where many people disagreeing (or fewer disagreeing strongly) doesn’t count against a comment as long as there’s no objective problem with the phrasing/on-topicness/etc. Realistically it seems as if without another outlet, some fraction of disagreement will get funneled into the flag system (even just through people looking hard for technically valid reasons to flag) even if you try to discourage it.

                                                                                                                        Just as a random thought, you can offer downvotes without treating them as -1 upvote (-0.5 or -0.25 or whatever in a ranking algorithm), or disagree flags that you treat differently from others.

                                                                                                                        tl;dr if I guessed at possible tweaks they might be allowing [no-reason] downvotes or disagree flags that have less impact than existing flags, just to keep that self-expression out of the flagging system; look for a way to factor in that being more-read will get your comments more negative attention; and maybe there’s something to do with the message wording, though it’s clearly already had a good amount of thought put in it.

                                                                                                                        1. 16

                                                                                                                          A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

                                                                                                                          Yes, I very much agree with this. On Reddit and Hacker News, which lobster.s is modelled after, I think of as:

                                                                                                                          • upvote: “I would like this to appear higher on the page”
                                                                                                                          • downvote: “I would like this to appear lower on the page”

                                                                                                                          Flag is a totally different thing than downvote – it means that the user is abusing the site, and continuing that behavior SHOULD result in getting banned from the site.

                                                                                                                          But expressing minority opinions (thoughtfully) may result in downvotes (*), but it should NOT result in getting banned or encouraged to leave.


                                                                                                                          As far as I can tell, the burntsushi warning was a result of mixing up these signals. I only read a portion of the Rust threads, so I haven’t followed everything, but from what I understand he was probably just expressing some opinions that people disagreed with, not actually abusing the site.

                                                                                                                          So I think you have diagnosed it correctly – the lobste.rs UI is sort of mixing these 2 signals up by omitting downvotes. The “flag” button has 5 reasons, and “off topic” and “me too” should be a downvote, not a flag IMO.

                                                                                                                          Again, the difference between the two is whether continuing the behavior should lead to a ban, and that’s a very important distinction.

                                                                                                                          (Honestly I am surprised that I haven’t gotten this banner ever, given that there is a minority of people who disagree with my opinions on shell. Since I’m writing a new one, I’m opinionated about it, but also invite disagreement)


                                                                                                                          (*) Some people may want opinions they disagree with to appear lower on the page, and some people might not. In my view it’s up to person to exercise that discretion – that’s why you get one vote :)

                                                                                                                          1. 3

                                                                                                                            upvote: “I would like this to appear higher on the page” downvote: “I would like this to appear lower on the page”

                                                                                                                            That’s not actually what they mean on Reddit:

                                                                                                                            Vote. If you think something contributes to conversation, upvote it. If you think it does not contribute to the subreddit it is posted in or is off-topic in a particular community, downvote it.

                                                                                                                            ‘I would like this to appear higher/lower on the page’ is so ambiguous that it leads to different people misunderstanding it and applying it differently.

                                                                                                                            1. 6

                                                                                                                              I’d say that’s a prescriptive definition, but a descriptive one is more useful [1], and thinking about it that way will be more likely to solve the problem.

                                                                                                                              That is, “users don’t read documentation”, and it means whatever the users think it means, and it changes over time. FWIW I’ve been using Reddit since 2007, and HN since 2011 or so, and there’s never complete agreement on what these things mean.

                                                                                                                              I’ve seen the debate about downvote vs. disagree several times over the years on the sites. I choose to sidestep it with the vague definition – that’s a feature! :) It’s OK to be vague sometimes.

                                                                                                                              In cases like this, where the number of flags leads to either banning or encouragement ot leave, I think it’s better not to be vague, and have 2 distinct mechanisms. This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

                                                                                                                              [1] https://amyrey.web.unc.edu/classes/ling-101-online/tutorials/understanding-prescriptive-vs-descriptive-grammar/

                                                                                                                              1. 2

                                                                                                                                This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

                                                                                                                                I don’t think it had both, rather the flag button just looked like a “downvote” button. You still selected from the same reasons that flag presented. This change is the one that altered the styling I believe. There’s plenty of places that refer to it as “voting” in the code still, but unless I am mistaken there was only ever the one mechanism.

                                                                                                                          2. 7

                                                                                                                            A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

                                                                                                                            So, the purpose of downvotes was emphatically not to express disagreement…it was to signal one of a handful of error conditions (bad information, factual inaccuracy, etc.). There is very little utility in seeing how many people merely disagree about something, especially given how fast people are to be mean to each other.

                                                                                                                            As a historical point, @jcs at one time disabled downvotes. It did not last more than a week or two before it was brought back.

                                                                                                                            1. 11

                                                                                                                              Sorry, I see how it was worded unclearly. I understand that the intention currently is to have no “disagree” button. I’m saying that having that button could reduce how often people flag comments they disagree with as having one of the error conditions, which they might do by by stretching definitions, picking a random reason, or just preferentially noticing real flaggable issues when they happen to dislike the content. (I think the unclarity comes from me using “downvotes” to mean “a no-reason downvote button” as opposed to flags as they exist–I edited to clarify above.)

                                                                                                                              It may be useful to let people push a button for disagreement even if your algorithms assign a low weight to those clicks. I say low weight rather than zero because having a 100%-placebo button in the UI raises other issues, even if moderators give zero weight to an easy expression of disagreement. I’d probably give a no-reason-given downvote nonzero weight in my head but I don’t expect others to adopt my weights.

                                                                                                                              Shorter, the idea is that people sometimes find a way to “downvote” for disagreement however you set up your UI, and you might get a net harm reduction by just giving them the button they want and then deciding what to do with the data.

                                                                                                                              1. 6

                                                                                                                                Ah, so kind of like the buttons on crosswalks?

                                                                                                                                Anyways, thank you for elaborating further. :)

                                                                                                                                1. 4

                                                                                                                                  Very well said. Even if the system is not equipped with a “disagree” button, some users may desire one and use whatever means are at hand to express themselves. The phrase, “the purpose of a system is what it does,” comes to mind.

                                                                                                                                  If existing mechanisms are not meant for use in disagreements, then we might ask if that is actually how they are being used.

                                                                                                                                2. 4

                                                                                                                                  That’s nice, but you can’t make users behave the way you want them to behave, if you’re not going to actively act against it. People not only downvote because they disagree, but they’ll also flag things as spam or off-topic or “troll” just because they disagree, or feel you’ve insulted their particular religious (as in vimacs-vs-spaces) beliefs.

                                                                                                                                  1. 6

                                                                                                                                    I think I’d prefer a system that discourages misuse of flags. Maybe a similar message for people who flag a lot

                                                                                                                                    1. 3

                                                                                                                                      Sure, folks determined to game things would still flag. My theory is decently many who flag today are not determined to game things, and would click a down arrow for “just no” given the option. That’d reduce noise in the flag system even though it wouldn’t eliminate it. (And not saying you couldn’t do other changes too.)

                                                                                                                                  2. 4

                                                                                                                                    in fact, you can have placebo downvotes that show up for the person doing the downvoting but do not affect the vote count otherwise. that would keep (almost) everyone happy, i think.

                                                                                                                                  3. 6

                                                                                                                                    Agreed.

                                                                                                                                    I think it’s fair to discuss whether the wording of the warning message can be improved, but various comments here that pretty much amount to “person X is popular, he should be exempt from the rules” are just tiring.

                                                                                                                                    1. 8

                                                                                                                                      How is being flagged a rule? It’s not a rule. That is the whole problem.

                                                                                                                                      Anyone can flag any comment any time.

                                                                                                                                      1. 2

                                                                                                                                        Receive too many flags, get a notice. That’s the rule.

                                                                                                                                        1. 9

                                                                                                                                          That is not a rule in the sense of rule that can be followed by users. I did’t read a single comment suggesting that shouldn’t apply to some user because of its popularity, but rather many people pointing out that this is a clear case illustrating that such system is broken and should be fixed for all users.

                                                                                                                                    2. 5

                                                                                                                                      rather than continue to engage in a community they don’t fit into

                                                                                                                                      Is lobster a community that accepts a plurality of opinion? If it is, then you will always have people disagreeing with you and therefore be liable to be reported by some of these people.

                                                                                                                                      Democracy leads to lynching. Mass reporting is basically the lynching of internet persona. Any kind of automatic penalty that can be initiated via reporting is simply giving power to the angry mob. Autoban and autosuspend based on reports have always been abused.

                                                                                                                                      1. 3

                                                                                                                                        You’re still here, so evidently the suggestion doesn’t apply to you. You’re glad to contribute.

                                                                                                                                        Why not show the warning to flag-happy people? I don’t have stats but I feel like that could be a better indicator of who doesn’t want to be here.

                                                                                                                                      2. 21

                                                                                                                                        Why do we need this passive aggressive banner anyway? It looks bad for Lobster, not for the person seeing it, and it’s not clear what problem it’s supposed to solve.

                                                                                                                                        1. 19

                                                                                                                                          We consistently see the vast majority of bad behavior come from the same few users. There’s a phenomenon, called the normalization of deviance, whereby people whose behavior is far away from the norm often genuinely don’t realize just how far it is. The banner, and the histogram that it links to, are intended as a way for people to self-correct. I personally am a big believer in giving people opportunities for redemption, and the histogram makes the point in a way that mere words can’t.

                                                                                                                                          1. 6

                                                                                                                                            far away from the norm

                                                                                                                                            Far away from the norm doesn’t mean bad. You seem to be assuming that it is.

                                                                                                                                            1. 8

                                                                                                                                              I appreciate you raising that. On the contrary, I agree that sometimes people’s behavior is far from the norm for positive reasons. I do think that, even in those cases, everyone can benefit from a heads-up and a chance to reflect about whether it’s what they want.

                                                                                                                                              1. 3

                                                                                                                                                As in reflecting on do you want to be good when everybody else is bad?? Am I reading that right?

                                                                                                                                                1. 7

                                                                                                                                                  Well, yeah. I know for a fact that people have left Lobsters for that reason (or at least, that’s how they see it). While I am sad when people leave, I don’t think anyone is obligated to stay somewhere that doesn’t make them happy. It would be wrong of me to try to trick or coerce people into staying when their free choice, with all the facts available, would be to leave.

                                                                                                                                                  I’m not really sure why you’re asking this, though. The flagging system doesn’t have the ability to determine who’s right and who’s wrong. Even if I wanted to take my personal concept of “good” and write code that ignores flags that aren’t “good”, there is no existing technology that can do anything like that. If that’s what you’re advocating for, feature request noted but I’m not able to do it.

                                                                                                                                                  1. 3

                                                                                                                                                    I’m not really sure why you’re asking this, though

                                                                                                                                                    I am not asking anything. I was commenting on an implicit bad argument you made (the implication that the norm is correct). You decided to double down by arguing that even if somebody was correct, they should ‘reflect about whether it’s what they want’. I simply then pointed out that this doesnt reinforce your original argument, because it itself is bad, because having a bad feature (aggressive warning about being reported) is not validated by having that feature also be useful to do useless thing (letting people reflect on whether they should be good, because the answer is always yes).

                                                                                                                                                    I don’t think anyone is obligated to stay somewhere that doesn’t make them happy.

                                                                                                                                                    Of course not. The point is that assuming that lobsters want a community that engage in meaningful discussion, then it should make people want to stay and be happy, that is a worthy goal. all of this is following your argument for the banner, the argument which I consider to be invalid, as pointed out.

                                                                                                                                                    1. 5

                                                                                                                                                      Being distant from the norms can hold little moral judgment and yet be relevant when operating in a constructed community.

                                                                                                                                                      1. 2

                                                                                                                                                        Okay, I guess that’s fair.

                                                                                                                                                        I think we disagree about something in there, but it sounds like figuring out what isn’t a high priority for either of us? I appreciate the discussion, anyway.

                                                                                                                                                2. 1

                                                                                                                                                  I at least didn’t read that implication into the post. Not all ways of being far from the norm are likely to result in seeing the banner. It isn’t a perfect system but it’s definitely not symmetric, so that assumption isn’t necessarily required.

                                                                                                                                                3. 2

                                                                                                                                                  Thanks for clarifying. But I still think it doesn’t make Lobster looks good because this feature implies that “deviant” behaviour should be corrected and can be automatically detected, and I expect most people (especially in tech) aren’t comfortable with either of these.

                                                                                                                                                  I’m wondering what’s the worse that could happen if the banner is not displayed? Maybe the person who’s being flagged all the time will get the point and adjust their behaviour, or maybe they don’t need to adjust anything and it’s other people that will start seeing that perhaps that person has a point. I also believe in giving people a second chance but I don’t think the solution to everything has to be algorithms, sometimes simple human interactions are preferable.

                                                                                                                                                  1. 4

                                                                                                                                                    The hypothesis that motivates the banner and histogram is that, by displaying them, those conversations about what to adjust will happen sooner and be more productive. The goal is to head off situations where somebody doesn’t believe mods when we tell them something needs to change, and they ultimately leave the site. It may be right or wrong but that’s the motivation.

                                                                                                                                                  2. 1

                                                                                                                                                    Do you think that the banner has an observable positive effect? Do you see a change in behavior from the few users you say produce the majority of bad behavior? As friendlysock says above, they see the banner nearly all the time. Do you think they have been falsely-flagged by this system? If so, can you estimate how high the false-positive rate of showing this banner? If friendlysock has been appropriately warned by this system, have you seen any change in their behavior onsite?

                                                                                                                                                    If it is truly only a few users producing undesirable behavior, could it be more effective to simply reach out to those users personally?

                                                                                                                                                    1. 4

                                                                                                                                                      I’m not going to publicly discuss my personal feelings about specific community members. It wouldn’t be fair to anyone.

                                                                                                                                                      I don’t have a methodology to conclusively determine how much influence the banner has had, but the site has a lot less flagging in general than it did a couple years ago. I take that to be an indicator that things have calmed down significantly, and I do think that’s good.

                                                                                                                                                      I do reach out to people personally from time to time, as do the other mods. The banner and histogram were created to address a shortcoming in that approach, where sometimes people aren’t willing to listen to criticism unless they’re presented with data that validates it. I’m skeptical of going fully data-driven for everything, but I think it’s nice to have at least this one particular metric for those who find it easier to take concerns seriously when there’s a metric.

                                                                                                                                                4. 20

                                                                                                                                                  Additional suggestion to all the previous improvements: make that banner a thing that’s moderator-initiated. I of course don’t have statistics on how many people see it (I never got it, so I just learned about it now, anecdotally) but I could imagine that this is rare enough that moderators could just evaluate sending this message to users by hand after the system flags the user internally.

                                                                                                                                                  1. 6

                                                                                                                                                    I of course don’t have statistics on how many people see it

                                                                                                                                                    I don’t have that statistic either, but as I posted elsewhere, only 13 people have been flagged more than me in the last month and I haven’t seen it. Only 7 people have been flagged 10 or more times. I’d therefore expect that trimming it down to 5 people that moderators need to take a look at per month is quite feasible.

                                                                                                                                                    If even that is too much of a load for the moderators, then it might be possible to select a dozen people who have been active recently but not participated in any of the flagged threads to do a triaging pass. At that point, having someone reach out to the person and say ‘you’ve had a lot of things flagged recently, is there anything going on in your life that you want to talk about?’ would be a fairly low load.

                                                                                                                                                    1. 3

                                                                                                                                                      You can a chart that gives you a clear idea how many people see it at /u/you/standing. So you can see it at /u/david_chisnall/standing and I can see it at /u/hoistbypetard/standing. The red Xs should represent users seeing the warning.

                                                                                                                                                      I’d never suggest that someone else should just add a thing to their plate, but it does seem like the kind of thing where, if there were a few active moderators, requiring someone to approve the red box might be an improvement for a relatively low effort. That feels like the kind of statement I should only make if I’m accompanying it with a patch, though, and I can’t offer that right now.

                                                                                                                                                  2. 15

                                                                                                                                                    Yep, I think the current wording will probably trigger exactly the wrong people “have I overstepped a border? Probably” and the people who are actively engaging in a little bit of flaming won’t be bothered.

                                                                                                                                                    1. 7

                                                                                                                                                      Following on the “let’s get shit done” vibe, is there an open issue for discussing this in more detail on github? I did a quick lookup and didn’t find any, would be cool if it was posted here (even better if as a top comment)

                                                                                                                                                      1. 5

                                                                                                                                                        Good idea, thanks! (Obvious in retrospect, like many good ideas.) I created an issue.

                                                                                                                                                      2. 4

                                                                                                                                                        I mildly agree with both of those things, but I also think that the suggestion further down the thread to re-introduce downvoting as “non-bannable feedback” could address the issue more directly.

                                                                                                                                                        https://lobste.rs/s/zp4ofg/lobster_burntsushi_has_left_site#c_padx5h

                                                                                                                                                        Some people may misuse flags and downvotes, but overall I think these guidelines are intuitive and easy to remember:

                                                                                                                                                        • upvote: this should go higher on the page
                                                                                                                                                        • downvote: this should go lower on the page
                                                                                                                                                        • flag: if the user continues this behavior, they should eventually be banned
                                                                                                                                                        1. 2

                                                                                                                                                          In my post I’m trying to focus on an easy thing to do to help reduce this problem. I have the impression that rewording the banner is something that’s reasonably easy, while “let’s reintroduce downvotes” or “the banner should be approved by moderators before being shown” are important change in process that require a lot of discussion. They may be excellent ideas, but it’s a lot more work to discuss and turn them into action, so I think it’s best not to mix the two kind of suggestions if we want to change anything at all in the short term; they should be discussed separately, and maybe differently. (I could just send a PR to rephrase the banner; any of the other suggestions requires more discussion with contributors, etc.).

                                                                                                                                                          1. 1

                                                                                                                                                            Yes, it makes sense to do the easy things first. My impression was that we had downvotes several months ago, so it shouldn’t that big a change to restore them, but I’m not involved so I could be wrong.

                                                                                                                                                            I was mystified by the removal of downvotes, and never saw the rationale for it. I saw one thread discussing it 3 or 6 months after they were removed, which still didn’t answer my questions about why they were removed. It made the site worse for me, but not enough to really complain about it. I know this is an all-volunteer effort and I didn’t have time to look at it myself.

                                                                                                                                                      1. 3

                                                                                                                                                        I generally like the runtime-level decisions Go makes, e.g. instead of object headers there are “fat pointers” (interface values) instantiated specifically when you want to do something dynamic, the use of interfaces rather than hierarchies. Despite the throughput issues vs. other GC’s, I like that Go’s allows internal pointers, avoids stopping the world for long, and doesn’t require a read barrier.

                                                                                                                                                        One cool thing Erlang/BEAM did that Go didn’t is separate shared and per-process local/private heaps at the runtime and language level.

                                                                                                                                                        In a language like Go that would probably involve a type qualifier like shared, and when something needed to be shared that wasn’t yet (or something shared needed to become private to a thread) you could copy it, with the compiler sometimes able to move where an allocation is done as an optimization. (There’s a loose analogy to creating pointers vs. values and stack/heap.)

                                                                                                                                                        It would help with a couple things. Most important, it’s a path towards safer concurrency. There are various approaches: Rust-like rules that accesses to the shared heap must be explicitly guarded, something more implicit (with some risk of locking not working how you meant), or at least patch up the ways race conditions cause type and memory unsafety today (e.g. write type/pointer and length/pointer pairs to the shared heap with atomics like x86’s cmpxchg128) like the JVM does. If you don’t build static concurrency safety into the language, a shared qualifier could at least possibly help static analysis and let dynamic checkers slow down fewer accesses. (If this sort of stuff sounds interesting you might like “smaller Rust” blog posts (1, 2) though it’s definitely its own idea only loosely related.)

                                                                                                                                                        It would also open up some options for GC. With a rule that shared data can’t point to any thread’s private data, you could revive Go’s ‘request-oriented collector’ idea (quick collection of data private to one thread when it quits), do per-thread collections that don’t have to worry about concurrent accesses, or even do moving or generational collection for local data. All that works because stopping one thread isn’t stopping the world; threads pause all the time. You could keep Go’s existing non-generational approach on the shared heap, but its performance could benefit when local allocations no longer factor into global GC rate. (Or you could go for a design like Java’s ZGC with a read barrier, but man, seems even harder than what Go does!)

                                                                                                                                                        I realize these things get vastly more complicated once you get into details. (How do you not get your lunch eaten by private<->shared copies and synchronization accessing the shared heap? What on earth is that “something more implicit” to semi-safely access shared stuff? etc.) And Go is Go and BEAM is BEAM and you can’t just order up a mix of the two. But the bulk of the Go runtime model with an explicit shared/private distinction tacked seems like a neat spot in the design space that I haven’t seen explored. If there are existing examples I don’t know of it would be neat to hear about ’em!

                                                                                                                                                        1. 3

                                                                                                                                                          I haven’t read about it in a while, but what you’re describing sounds similar to the way references work in Pony. They have lots of great papers about how their GC works, but here’s a start to the different kinds of references: http://jtfmumm.com/blog/2016/03/06/safely-sharing-data-pony-reference-capabilities/

                                                                                                                                                          1. 1

                                                                                                                                                            Thank you!

                                                                                                                                                          2. 3

                                                                                                                                                            (Pony contributor here) Like @Pentlander says, Pony checks several of these points:

                                                                                                                                                            • It enforces write-uniqueness (among other things) across actors using reference capabilities, which allows you to “move” data across actors without any copying–most things are pass-by-reference.

                                                                                                                                                            • Each actor has its own heap, so GC can happen independently.

                                                                                                                                                            There’s a talk comparing Pony to Erlang here: https://www.youtube.com/watch?v=_0m0_qtfzLs if you want to take a look. If you have any questions, you can post them here, or take a look at the community page, join, and asks question there!

                                                                                                                                                            1. 1

                                                                                                                                                              Thanks!

                                                                                                                                                          1. 1

                                                                                                                                                            Cool to know that tf exists–I feel like data everywhere ought to be easier to query in more complicated ways and this seems like a contribution to that.

                                                                                                                                                            An approach to cutting garbage when streaming in Go is to reuse your buffers and minimize copies. In this case, instead of calling ReadBytes, which allocates a byte slice, you might call Scanner.Bytes, which is, underneath, reading blocks into the same []byte buffer over and over and returning slices pointing to one line in that slice. Then you have to make sure you don’t retain slices across reads, and copies/allocations later in the processing chain can still bite you. I don’t know how it would have worked out in this context, and might be especially tricky to make work with the parallel stuff.

                                                                                                                                                            Also, if tf doesn’t have much live heap data, Go will GC frequently to keep the heap small, when you might rather save some CPU by using more RAM. That’s a long discussed issue worked around with tricks or high GOGC.

                                                                                                                                                            I’m not saying tbray should have done this stuff, nor am I trying to draw larger conclusions about the speed or goodness of Go or anything like that. Just noting this for any practical interest if you this kind of thing in Go.

                                                                                                                                                            1. 3

                                                                                                                                                              The overhead of the kernel storage path accounts for half of the access latency for new NVMe storage devices.

                                                                                                                                                              It’s clearly shown early in the paper, but bears noting that does not refer to Flash SSDs, which start at ~100µs latency. Rather it’s talking about Intel Optane SSDs, which respond much faster but have around 10x the cost per GB (just quickly comparing datacenter-class SSDs of both types).

                                                                                                                                                              It would be fun if more of us could at least expect to be dealing with lower-latency NVM soonish, but even that seems uncertain, as Intel ended its consumer Optane offerings and Micron, who owns the sole fab, wants out of that business. Anything could still happen, of course.

                                                                                                                                                              Still cool to see approaches to these problems, just didn’t want the “kernel causes half of my latency” premise to leak into contexts where it doesn’t apply.

                                                                                                                                                              1. 4

                                                                                                                                                                Relevant to the title, these huge language models are in effect part complex new technology and part…search index. They memorize phone numbers and addresses from the Internet. GPT-3 generates plausible-sounding news articles about gay marriage in the United Methodist Church, but it also seems to have ingested actual news articles about the topic (not the best source but see here). Typing questions into Google search also gets you plausible looking text, but there you (hopefully) know where it came from!

                                                                                                                                                                I’m not saying there is nothing new or remarkable with big language models, just that how impressive the results are is not a direct measure of how much generalization they’re doing, as separated from having relevant stuff in the training data and doing a good job stringing it together. Although it’d be a weird loop-around in the history of AI, a general DNN that had built up the ability to do arithmetic (other than from memory), or answer the kinds of questions old-school expert systems did (‘are camels generally bigger than ants?’), could be impressive in a totally different way from what we have.

                                                                                                                                                                1. 7

                                                                                                                                                                  Really good one. I got interested in Go immediately in 2009, contributed pre-1.0, been using it professionally for 6 years now, and still learned two things here: about the iota counting behavior, and that reading from nil maps is ok.

                                                                                                                                                                  As to something that could possibly be added to the article, the best pitfall of Go that comes to my mind is the value-receiver data race, as described by Dave Cheney. Took me a long while to grasp what’s going on, and is really subtle.

                                                                                                                                                                  1. 1

                                                                                                                                                                    Is it because the channel is getting copied?

                                                                                                                                                                    1. 3

                                                                                                                                                                      I think it’s that compute() ‘owns’ rpc and might be writing to it at the same time the following line is (implicitly) copying rpc onto the stack to serve as a value receiver. It’d still be a race if the sync stuff were located outside the struct; the only important thing is something might be messing with the struct in another thread while you call one of its methods that has a value receiver.

                                                                                                                                                                      It’s not a race when version() has a pointer receiver only because version() doesn’t access anything that compute() is messing with (since it doesn’t access anything at all!).

                                                                                                                                                                      1. 2

                                                                                                                                                                        Yep, the sneaky thing is that calling a value-receiver method is implicitly a “read” of the receiver object due to the need to copy it (even if the receiver value is ostensibly ignored, as in this case, a copy is apparently still being made). I spoiled it somewhat by how I titled the link; if I saw such code in a code review before reading this article, I would not connect the dots and realize that calling a value receiver triggers an implicit read from data race point of view.

                                                                                                                                                                      2. 2

                                                                                                                                                                        I found this explanation on the golang-nuts list: https://groups.google.com/g/golang-nuts/c/gJtXQhpoA4U/m/Atlm2IHVCAAJ

                                                                                                                                                                    1. 2

                                                                                                                                                                      I’ve only played with Rust, but I guess I can talk about getting started with it going on that.

                                                                                                                                                                      Like the commenter there I was a bit bemused by the statement that Rust isn’t a beginner language. It’s a huge language, but The Rust Programming Language tries to minimize use of unnecessary programmer jargon and to step through topics and examples in detail. You do need to go through the book slowly, even if you are used to picking up new vaguely-C-ish languages quickly. I would rather start off learning Rust than some other systems languages, compiler errors and all.

                                                                                                                                                                      I do think a nice editor setup is handy when getting started; I liked VSCode with either the Rust plugin or rust-analyzer (which is newer with neat features like displaying inferred types live, but was slightly glitchy when I tried it). Autocomplete looks up some of those method names and so on that you don’t know yet, and the compiler’s helpful error messages are even more helpful right there in the editor. It’s handy to get used to searching around docs.rs and crates.io, and sometimes play.rust-lang.org can help you resolve confusions about details of the language. Also, users.rust-lang.org seems to be the most used question-and-answer site. Clippy(!) is a linter and seems particularly helpful for getting conventions and idioms down (I admit I didn’t use it much).

                                                                                                                                                                      Though Rust folks do amazing work to make the language approachable, it’s still nontrivially more work to write a program when when it has to also contain the raw material for a proof that memory use and concurrency are correct, and when so much detail is made explicit. A question I have about the next years is how much that difficulty is addressed through Rust folks just continuing to chip away at ergonomics bit by bit, or whether more progress happens through some other language finding an intermediate point (see also) that errs towards safe and efficient but isn’t quite zero-overhead, full-control, and explicit like Rust. Obviously we have lots of other workable compromises out there (daily Python user here!), but I think the surprising usefulness of Rust in many domains suggests there is some fertile design space around it.