1. 6

That sounds like base rate fallacy, so now I’m doubting whether I understand either of them :)

1. 6

This one is also useful to visualize: https://byrdnick.com/wp-content/uploads/2020/07/Base-rate-box-problem-drug-test-diagnostic-reasoning-nick-byrd-512x446.jpeg

In the base rate fallacy, the correct interpretation is a straight-forward application of Bayes’ rule, which turns out to be unintuitive for the lay-person. Essentially, even if you have a very accurate test or detector, when you have a very low “base rate” or true prevalence of the condition in the population, the results of your test can turn out to be uninformative.

1. 3

The Wikipedia article on Simpson’s paradox does a good job of illustrating it with examples. If anything, it’s a caution (beyond those which already exist) against reading too much into base rates.

1. 12

It can be useful and intuitive to illustrate the paradox in a chart. For example: https://twitter.com/remiemonet/status/984893903605321729?s=20

If we look at the population as a whole, the best fit line points in a certain direction. If we break down the same data into key groups, it’s clear that the best fit line for each group would point in a completely different direction. Our interpretation of the data can be different or even reversed, depending on whether we include group identification as a feature.

1. 4

There was an attempt.

But after this:

This is where I call bullsh*t. I have never had a static type checker (regardless of how sophisticated it is) help me prevent anything more than an obvious error (which should be caught in testing anyway). What static type checkers do, however, is get in my way. Always. Without fail.

Discourage himself.

And all this brag, what s-expressions is only trees. What about: `[ ] ‘ # :` ?

1. 9

Honestly, if a type checker gets in your way either the type checker is too simple or you have an attitude problem. Sometimes both

1. 4

Or you are doing something really, really weird.

1. 2

I have a question for you or anyone else interested. I’m not a type-system skeptic… I cut my teeth on Ocaml and Haskell as a language hobbyist. Lately I’ve been using more Python and I’ve been wondering how to get a specific function to compile. In dynamically typed languages it’s actually very common to come across a very generic “find” function over lists which takes an object to look for, the list, and optionally takes a key function (which produces the comparison object from each item in the list) and a test function (which defaults to the equality operator). This is really handy because the less names I have to learn (i.e. both `find` and `findBy`), the better. Naturally I wanted to see if I could do this in a statically typed language (without dependent types… those are hard for me. I know you’re the formal methods guy around here but forgive me for being recalcitrant about learning new tools).

I like OCaml, so I took a crack at it in OCaml but can’t get it to be as polymorphic as I want it to be. Here’s a simple version.

``````let rec find (o : 'b) (xs : 'a list) (key : ('a -> 'b) option) (test : ('a -> 'a -> 'bool) option) =
match xs with
| [] -> None
| x :: rest ->
(
match key, test with
| Some(k), Some(t) ->
if (t o (k x)) then (Some x)
else find o rest (Some k) (Some t)
| Some(k), None ->
if (o = (k x)) then (Some x)
else find o rest (Some k) None
| None, Some(t) ->
if (t o x) then (Some x)
else find o rest None (Some t)
| None, None ->
if (o = x) then (Some x)
else find o rest None None
);;
``````

I’d like to be able to call this like `find 2 ["my"; "name"; "is"; "kris"] (Some String.length) (Some (=))`. Unfortunately Ocaml gives me back `val find : 'a -> 'a list -> ('a -> 'a) option -> ('a -> 'a -> bool) option -> 'a option = <fun>` It’s been a while since I worked through a Hindley-Milner example but I think this is the correct refinement because: what if the needle and the haystack items are differently typed, but there’s no key or test function? The default is structural equality, which is `'a -> 'a -> bool`, so it follows that the needle must always have type `'a`. I tried a few different things, like breaking the match arms into different functions, but I’m not sure what to do. One thing I haven’t tried yet is trying this in Rust and using the `Any` trait. Specifically, how can I, or what is the closest I can get to, have one function I can call to do all the different kinds of list searching that I want to do?

I don’t think this is “weird”. In general there are a lot of examples that make command line interfaces, dynamic languages, etc. feel so ergonomic, and it’s also why I feel like the answer might lie in dependent types: there are a lot of times where a function or verb is parameterized by the types of the arguments or the way that you call it. I suspect a hypothetical language that supports overloading by keyword argument would suffice for this example but there are other examples I have where there’s a similar tradeoff between completeness and ergonomics. But furthermore, this isn’t really an ad-hoc polymorphism example, because I can write a single definition in dynamic languages. Something like

``````find (o, xs, key=None, test=None):
if len(xs) == 0: return None
else:
x = xs[0]
if key: x = key(x)
t = default_equality
if (test): t = test
if test(o, x): return x
else: return find(o, x[1:], key, test)
``````
1. 4

Specifically, how can I, or what is the closest I can get to, have one function I can call to do all the different kinds of list searching that I want to do?

Have a find function that takes a predicate for an item, and call it with different predicates. This is how `List.find` already works in `Base`: depending on how you call it, you can search for a needle in a haystack:

``````List.find haystack ~f:(My_type.(=) needle)
``````

You can search for some nested key, of a different type:

``````List.find haystack ~f:(fun x -> Other_type.(=) x.key needle)
``````

You can find something using something other than equality:

``````List.find haystack ~f:(fun x -> Shape.fits_inside x.hole shape)
``````

I get what you’re asking, though. I don’t mean this to be snarky or annoying, just to say that this isn’t… the thing that works well in Python doesn’t really work well here.

I think one difficulty is that you’re using the polymorphic equality function, which is… basically shouldn’t really exist. It has surprising (illogical) behavior in the face of any nontrivial abstract type; it will throw exceptions if you pass it certain types of objects; it just shouldn’t exist.

So let’s pretend that we have no polymorphic equality. Then how do you write that function?

Well, now you need to pass an explicit comparator. Basically, make `test` a required argument. Okay; no problem. You wind up with a signature like this:

``````let find (needle : 'a) (haystack : 'a list) ?(key : ('a -> 'b)) (test : ('b -> 'b -> 'bool)) = ...
``````

I’m making `key` a named optional argument instead of an explicit positional `option` because that’s more stylistically natural, but it’s equivalent here.

But what’s the default value of `key`? Easy: the identity function. We want it to default to `Fn.id` if it’s not provided explicitly.

Except… that doesn’t typecheck. That doesn’t actually make sense. `Fn.id` isn’t `'a -> 'b`; it’s `'a -> 'a`. This makes sense if my `test` predicate expects an `'a`, but the type signature says right there that it accepts a `'b`. What if I call it with a differently typed `test`, but don’t provide a `key` argument?

You kind of want to say “you can’t do that.” In other words, make `key` required when `test` has a different type, and allow it to be optional when `test` doesn’t. But you can’t express that – you can’t decide that an argument is optional or not, based on the type of the function passed to `test`.

And you know all this, which is why you’re bringing up dependent types. Can they let you determine, at runtime, whether this argument is optional or not?

Well, not in OCaml. But in some other type system? Sure! Dependent types wouldn’t even be necessary: just moving the type-checking of the defaulted arguments to each callsite would get you the behavior you want (you’d have to use actual optional arguments, instead of defaulting them in a `match` statement in the function body, but whatever).

You can express the behavior you want in OCaml; you can construct predicates like this dynamically at runtime – but just not using optional arguments and a single function call. There are other ways to do it; heck, as soon as you make `key` a required argument, you have done it – callsites can typecheck whether `Fn.id` is valid for each individual invocation of your function. But that’s not the API you want.

I think this is an interesting illustration of a point the original author of this post was trying to make: why is the type system getting in my way like this? After all, Python lets you have the API you want. Why doesn’t OCaml?

Well, this example makes just as little sense in Python as it does in OCaml. Python just allows programs that use the “wrong” types. Want to compare a string and an int? Go ahead. False, I guess? Hope that was what you wanted; hope you meant to compare a string and an int. The Python function works if you use it right – if `o` is the same type that `test` expects, etc. But OCaml wants to ensure that all possible uses of the function are valid. It doesn’t want you to say “there is a right way to call this function,” it wants to say “all ways you can call this function are valid.”

These are very different goals! There are some contexts where you want that extra assurance, and some where you don’t. Depends on what you’re doing. (Er, sorry – what I meant to say was “static typing is pointless.”)

I have run into this probably before (specifically the “I want an optional argument to transform this value that defaults to the identity function”), and it’s annoying. I wish I could express that; I wish I could hold the compiler and soothe it and explain to it that it’s going to be okay. But instead I just make it a required argument and move on.

But in this case, you don’t even need to do that. The canonical OCaml way to do this is simpler and more flexible than the function you’re trying to write. You say you don’t think your function is “weird,” but it does look weird to me: why write this very specifically-shaped `find` function instead of the far more general one that lets you pass any predicate you want?

``````List.find haystack ~f:(My_type.(=) needle)
``````

The `key` example you want where the needle is the same type as elements in the list and you still want to compare it can be accomplished with the not-very-well-named `Comparable.lift`:

``````List.find haystack ~f:(Comparable.lift Identifier.(=) ~f:(fun x -> x.id) needle)
``````

It’s verbose and kinda ugly, but OCaml is a verbose and kinda ugly language :)

Last thing: you say you want a single function to do everything so you don’t have to remember which function to use. But why is `find` special? What if you want to, say, filter a list next? Do you have a `filter` function that takes these same optional arguments?

By breaking out the functions that construct these predicates from the functions that use the predicates, you end up with a (subjectively!) simpler program with less to think about. Find takes a predicate. Filter takes a predicate. Partially applying a comparator will get you a predicate. `Comparable.lift` is one way to compose comparators. String these together, and now any function that takes a predicate gets this compare-on-a-key function; you don’t need to go in and add these optional arguments to every function you want to call.

Okay I’m just rambling now and I have an undefined-is-not-a-function exception to track down so I’ll go now.

1. 2

I’m not an OCaml expert but I probably wouldn’t introduce `option` types into this signature. Your key and test parameters have reasonable “default” implementation that are as convenient as typing `None` or `Some`:

``````let rec find (o : 'b) (xs : 'a list) (k : ('a -> 'b)) (t : ('b -> 'b -> 'bool)) =
match xs with
| [] -> None
| x :: rest ->
if (t o (k x)) then (Some x)
else find o rest k t
;;

let itself x = x ;;
let always x _ _ = x ;;

find 2 ["my"; "name"; "is"; "kris"] String.length (=);;
find "my" ["my"; "name"; "is"; "kris"] itself (always true);;
``````
1. 1

I presented a simplified problem. The arguments are optional because I wanted to use optional arguments in OCaml so that I didn’t have to always provide a key or test function (the typical case). Furthermore the find function might take an arbitrarily large number of options, like fromEnd, so even though two optional arguments are simple enough to write explicitly, it would get unwieldy for more. Obviously the more power you give a function the less likely it is to be able to find a single signature for it. You can easily take this too far to the point of making a function with uselessly large surface area but I think there’s a grey area where a function is conceptually small enough to be useful and ergonomic for a lot of people but way too large to be statically typed.

1. 3

Well, you asked “Specifically, how can I [do the above]” and both I and ianthehenry gave pretty specific examples `:)`

I do agree with ianthehenry that it sounds like you want OCaml to be more like Python but there are some fundamental language design choices which mean that Python can do some “unsound” things that OCaml just won’t allow.

I think I get what you mean referring to large number of optional parameters. This seems to be a common pattern in Python, especially in the popular scientific and ML library APIs. `sklearn.svm.SVR` as an example has 11 options, and that’s not even a lot in comparison to other sklearn functions!

That sklearn API has more design issues than just having “too many” parameters. Two of those options are only valid when combined with certain choices for another option. Although this is documented, the language does nothing to help prevent usage mistakes here. Some of the options specify that they must be positive real numbers, which is a case where a stronger, dependent type system might come in handy.

Human errors in setting up complex options like this are common, and stupid mistakes like that are easy to make. This sort of bug is often much more subtle and pernicious than an error in logic. How will I know what will happen if I try to fit an SVR while invalidating some of those documented constraints? At a quick glance in the Python code, there’s no obvious runtime checking either, so who knows! How could unit testing solve this problem? Furthermore, why should the burden of testing be passed off to us end users, when we didn’t design this API?

But in a typed language this API would probably should be designed differently from the start. I would imagine some sort of Builder pattern might be appropriate, as it is well known as an alternative to having lots of constructor parameters. (Again, I’m not an OCaml expert so I don’t know the idiomatic forms.)

The typed language design tenets say to “make illegal program states unrepresentable.” That does mean that typed API design requires some more effort, to carefully model parameter variants rather than dumping a bunch of defaults into the function signature. In exchange for the cost of design, you get to make certain guarantees to your clients. This is what the typing proponents mean when they say typing “eliminates a whole class of bugs.”

2. 1

Thing is that I’m always doing something really really weird when I’m between states of working code, i.e. when I’m doing anything that I cannot instantly get right. I find that being in that dark zone is usually more fun and efficient with a less restricting language. A less restricting language kinda tries to work with me during the journey, but a stricter language just goes “dude, where the fuck are you even going?”

1. 1

I’d love to see an example of a language working with you here. I’ll admit my primary dynamic typing experience is Ruby where doing something invalid immediately results in an exception and (usually) a crash.

1. 1

Perhaps I’m thinking via negative examples when working with languages like Rust or Haskell. If I do something weird there, I just get an entirely unhelpful type error.

3. 4

I presume (though the author does little more than rant about this so I’m being charitable here lol) that the author’s position is that the contortions necessary to satisfy the type checker in cases where behavior is well understood is too high, and that guards/assertions/contracts around tricky parts of your code offer much of the same benefits as types without forcing the contortions around all of your code. I’m not sure how to test the veracity of either side of this claim, but I do think it has merit, having had to spend more time than I’d like building up State monads and lifting results out.

FWIW I still largely think this is a neurotype thing. I suspect folks who find abstractions easy/empowering will have little trouble juggling abstractions in their head and will prefer to use that rather than think about weird/silly edge cases. I suspect other folks prefer focusing on a more concrete approach in their code, and these folks will lean on things like contracts, property testing, and regular old unit testing for assurance. `sqlite` certainly offers us a window into what the latter approach could look like. I don’t really have the data to back up my thoughts on the matter though so I’m idly speculating as much as everyone else 🤷.

1. 3

Monad transformer stacks (besides being an advanced feature most code doesn’t need) aren’t a type system thing, if you want the same behaviour in a unityped language you’d need very similar machinery. Free monads are a cool technique, but same thing. I don’t use lenses often (and avoid `lens` especially) but maybe they are poorly enough set up that the type system fights you, I couldn’t say.

In general I find that if the compiler says I’m wrong, it is I who is wrong. The idea that “oh, this code would work but the dang type system won’t take it” is the thing I just can’t see. Sure, if you’re using advancing tools you need to understand the tools, but if you took away the type system all the compile errors would become equivalent runtime crashes. You’re not fighting the types usually, at most fighting yourself / your understanding of the tool. I definitely wouldn’t suggest most new Haskellers use monad transformer stacks (and only occasionally transformers at all… maybe ExceptT) or lenses, just as I wouldn’t suggest a new Rubyist reach straight for refinements or metaprogramming or a new rust programmer use Rc or RefCell

1. 3

unityped

This continues to be one of the most smug and contemptuous ways people refer to dynamic typing. Not as bad as “unethical” but up there

1. 2

How is it smug or contemptuous? I’m a Rubyist and have nothing against dynamic typing or anything like that.

2. 1

Monad transformer stacks (besides being an advanced feature most code doesn’t need) aren’t a type system thing, if you want the same behaviour in a unityped language you’d need very similar machinery. Free monads are a cool technique, but same thing.

Hm. I feel that there aren’t many benefits to using an ML type system if I can’t layer guarantees with a monad transformer stack or a free monad approach. Without those, I feel like I’m just working with any other somewhat strong type system. I may be throwing out the baby for the bathwater here since I haven’t used more “relaxed” languages like Ocaml or SML, mostly spending time in Haskell.

In general I find that if the compiler says I’m wrong, it is I who is wrong.

Sure and I don’t think that’s what TFA is talking about. I think TFA is saying that, the cost the author pays for using an ML-style type system is too high in instances of simple and clear logic to justify the benefits that it offers when the compiler tells you you’re wrong. This is a more complicated value tradeoff and I think it largely depends on the person. Knuth is famous for advocating to write prose to accompany logic in order, so the reader of code follows along as one does a math textbook and I presume this allows Knuth to keep the bugs in his code very low.

1. 2

I may be throwing out the baby for the bathwater here since I haven’t used more “relaxed” languages like Ocaml or SML, mostly spending time in Haskell.

Me too. I’ve played with SML and done some rust and swift, but mostly I use Haskell.

the cost the author pays for using an ML-style type system is too high in instances of simple and clear logic

Right, whereas I find for those cases (in Haskell anyway) I don’t need to even write any types at all. I just write the code as I would in Ruby (in ref to types, obviously syntax and approach will differ a bit) and that’s it. Types get inferred and if I make a typo or logic error I get a message when I run it with runhaskell just like I would when running the equivalent ruby script with ruby.

Anyway, I think maybe the issue is that it’s hard to discuss this in the abstract. Might be more constructive in the context of a particular fight someone is having, which is really a topic for pairing and not a forum, unfortunately.

1. 7

The best any static type checking will let you do is array[float]

Can’t Idiris at least do the “Monotonically Increasing Float” part?

1. 9

It can do the whole thing, because the dependent type system is capable of end-to-end formal verification. Any FM coding language can verify properties statically, even if it’s not in the type system, so you can do this in HOL4, SPARK, Dafny, etc. It’s just balls-hard (because formal verification is balls-hard in principle).

1. 12

It can do the whole thing, because the dependent type system is capable of end-to-end formal verification.

This effectively moves the complexity of expressing your invariants from one place (like tests) to another (a type system extensible by arbitrary code).

Type systems only feel magical as long as they stay simple and declarative. Which necessarily keeps them limited.

1. 3

Type systems only feel magical as long as they stay simple and declarative.

I wholeheartedly agree with this statement, but I doubt you’ll find any pair of programmers that agree on what’s simple and declarative. I bet if you asked around, you’d find that any point in the entire spectrum will be picked by someone as the sweet spot.

2. 2

Last I checked most of these end-to-end FV attempts require a lot of human intervention (i.e. a human has to prove the properties and encode it in the coding language). Has that changed?

1. 10

Nope, still sucks and I still hate it

(It’s gotten better in the past 6 years as people got better at SMTing all the things, but you still need a lot of human intervention to get anything interesting verified)

1. 1

Lean has some cool machine-assisted tactics like `suggest` and `library_search` that will offer help to find a theorem or proposition that will complete a proof, but these are pretty brute and still often require a lot of work. They’re hit-or-miss but when they hit it’s like magic.

3. 9

Every half-serious statically typed language can create new types `kilometers` and `miles` from float. Then you won’t confuse array[kilometers] and array[miles] and possibly prevent \$330 million dollars from being almost totally wasted (https://en.wikipedia.org/wiki/Mars_Climate_Orbiter).

That’s quite a bit better than array[float]. And you don’t need to use a research language to do it, either.

1. 3

Mars climate orbiter is a good example of why the phrase ’‘should be caught in testing” is not sufficient justification for not using additional checks and fail-safes. I assume NASA LM’s testing regimes are fairly rigorous (correct me if there is evidence to the contrary) but in some applications you need another layer of checking. Having said that I also assume that the software in question did have a type system, and it simply wasn’t used in the way you describe.

1. 14

The type system couldn’t help in this case: the data in US units (pound-seconds) which should have been in metric units (Newton-seconds) was read from a file on disk. The software reading the data from that file thought it was metric, but it wasn’t. Perhaps the file format should have included the units? But the file format as specified was only ever supposed to contain Newton-seconds, so why would they include the units?

Anyway, it was an interface problem, rather than a type system problem. See Mars Climate Orbiter Mishap Investigation Board Phase I Report

1. 3

In a way, the interface problem was a type system problem, but perhaps one we aren’t well equipped to solve across systems.

One could imagine attaching typed schemas to certain files, and writing language bindings that enforced that stored values must be strongly typed, with explicit units.

1. 3

Seems like modern type systems don’t solve this problem as c– mentioned. It is certainly true that no error checking method is 100% foolproof. My main point above was merely that testing isn’t either, and having multiple independent error checking systems is a good idea. Type systems are useful but should not be relied upon.

1. 1

Absolutely. It even goes beyond merely having multiple independent error checking systems: you have to have the processes, or organizational culture, in place to act accordingly. It’s interesting that the Mars Climate Orbiter problem was actually detected multiple times before the disaster occurred, but no-one quite took it seriously enough.

2. 3

This misses the payoff of things like type-classes, where you can get the language to provide implementations of all sorts of things. A simple example is deriving lexicographic ordering for tuples if the type of each member is ordered. As much as I enjoy editing in a sexp-aware editor, the idea that static types only constrain you is very outdated thinking.

1. 2

Even dispensing with sophisticated types, having a smart constructor that might fail at runtime but on success returns a wrapper type which guarantees it’s an “array of numbers with a mean value of such and such and a standard deviation of such and such” can help avoid many mistakes.

1. 2

There’s also “Category Theory for Programmers” by Bartosz Milewski, a good read:

https://github.com/hmemcpy/milewski-ctfp-pdf

1. 3

I have trouble with CTFP and most other Category Theory treatments because they seem completely ungrounded in the practical and concrete affairs of software development. It is as if a book on Relational Modeling for example spent all its text on defining set operations, tuples, projections, and joins, and all their associated theorems, but never delved into an example that illustrates how the theory applies in the real world of say a sales database. The theory needs motivating examples to be accessible to the lay-programmer!

1. 11

Ah.

My pet hobby horse.

Let me ride it.

It’s a source of great frustration to me that formal methods academia, compiler writers and programmers are missing the great opportunity of our life time.

Design by Contract.

(Small Digression: The industry is hopelessly confused by what is meant by an assert. And subtle disagreements about what is meant or implied by different programmers is an unending source of programmers talking passed each other).

You’re welcome to your own opinion, but for the following to make any sense at all, put aside whatever you mean by “assert” for the duration, and accept what I mean. You can go back to your meaning after we have finished discussing this comment.

By an assert in the following I mean, it’s a programmer written boolean expression, that if it ever evaluates to false, the programmer knows that the preceding code has an unknown bug that can only be fixed or handled by a new version of the code.

If it evaluates to true, the programmer full expects the subsequent code to work and that code will fully rely on the assert expression being true.

In fact, if the assert expression is false, the programmer is certain that the subsequent code will fail to work, so much so, there is no point in executing it.

So going back to DbC and formal methods.

Seriously. Writing postconditions is harder than just writing the code. Formal methods are way harder than just programming.

But we can get 90% of the benefit by specializing the postconditions to a few interesting cases…. aka. Unit testing.

So where can Formal Methods really help?

Assuming we’re choosing languages that aren’t packed with horrid corner cases… (eg. Signed integer overflow in C)…

Given a Design by Contract style of programming, where every function has a bunch of precondition asserts and a bunch of specializations of the postconditions……

My dream is a future where formal methods academics team up with compiler writers and give us…

• Assuming that every upstream assert expression is true, if it can be determined that any downstream assert will fail, the compile will fail with a useful warning.
• Where for every type, the programmer can associate an invariant expression, and the compiler will attempt to verify that it is true at the end of the constructor, and the start and end of every public method and at the start of the destructor. If it can’t, it will fail the compile with a warning.
• Wherever a type is used, the invariant expression can be used in this reasoning described above.

So far, you might say, why involved the compiler? Why not a standalone linter?

Answer is simple… allow the optimizer to rely on these expressions being true, and make any downstream optimizations and simplifications based on the validity of these expressions.

A lot of optimizations are base on dataflow analysis, if the analysis can be informed by asserts, and the analysis can check the asserts, and be made more powerful and insightful by relying on these asserts… then we will get a massive step forward in performance.

My experience of using a standalone linter like splint… is it forces you to write in a language that is almost, but not quite like C. I’d much rather whatever is parsed as valid (although perhaps buggy) program in the language by the compiler, is parsed and accepted as a valid program by the linter (although hopefully it will warn if it is buggy), and vice versa.

I can hear certain well known lobste.rs starting to the scream about C optimizers relying on no signed integer overflow since that would be, according the standard, undefined and resulting in generated assembler that results in surprised pikachu faced programmers.

I’m not talking about C. C has too much confused history.

I’m talking about a new language that out of the gate takes asserts to have the meaning I describe and explains carefully to all users that asserts have power, lots and lots of power, to both fail your compile AND optimize your program.

1. 5

As someone who has been using Frama-C quite a lot lately, I can’t but agree with this. There’s potential for a “faster than C” language that is also safer than C because you have to be explicit with things like overflow and proving that some code can’t crash. Never assume. Instead, prove.

1. 3

for the following to make any sense at all, put aside whatever you mean by “assert” for the duration, and accept what I mean. You can go back to your meaning after we have finished discussing this comment

Didactic/pedagogical critique: in such a case, it may be more appropriate to introduce a new term rather than using one which has a common lay meaning.

My dream is a future where formal methods academics team up with compiler writers and give us […]

Sounds a lot like symbolic execution.

1. 3

using one which has a common lay meaning.

Does assertions have a different meaning than the one given here?

1. 4

I have colleagues for whom it means, “Gee, I didn’t think that input from outside the system was possible, so I want to know about it if I see it in unit test, and log it in production, but I must still handle it as a possibility”.

When I personally put in an assert at such a point, I mean, “a higher layer has validated the inputs already, and such a value is by design not possible, and this assert documents AND checks that is true, so in my subsequent code I clearly don’t and won’t handle that case”.

I have also seen debates online where people clearly use it to check for stuff during debugging, and then assume it is compiled out in production and hence has no further influence or value in production.

1. 1

GTest (Google Test), which I’ve recently had to use for C++ school assignments, refers to this in their macro names as `EXPECT`. Conditions whose failure is fatal are labelled with `ASSERT`. This makes intuitive sense to me: if you expect something to be true you accept its potential falsehood, whereas when you assert something to be true you reject its potential falsehood.

1. 1

TIL! Thanks! I only knew about assertions from the contract perspective.

2. 1

A common design choice is that assertions are evaluated in test environments, but not production. In that case, plus a test environment that you’re not confident fully covers production use cases, you might use assertions for hypotheses about the system that you’re not confident you can turn into an error yet.

I’m not sure that’s a good idea, but it’s basically how we’ve used assertions at my current company.

3. 2

Alas, my aim there is to point out people think there is a common lay meaning…. but I have been involved in enough long raging arguments online and in person to realize… everybody means whatever they damn want to mean when they write assert. And most get confused and angry when you corner them and ask them exactly what they meant.

However, the DbC meaning is pretty clear and for decades explicitly uses the term “assert”… except a lot of people get stuck on their own meaning of “assert” and conclude DbC is useless.

Sounds a lot like symbolic execution.

Ahh, there is that black or white thinking again that drives me nuts.

Symbolic execution and program proving is a false aim. The halting problem and horrible things like busy beavers and horrible fuzzy requirements at the UX end of things make it certain that automated end to end program proving simply will never happen.

That said, it can be incredibly useful. It’s limited for sure, but within it’s limits it can be extraordinarily valuable.

Odds on symbolic execution is going to fail on a production scale system. Not a chance.

However, it will be able to reason from assert A to assert B that given A, B will fail in these odd ball corner cases… ie. You have a bug. Hey, thats’ your grandfathers lint on steroids!

4. 2

You might find the Lean theorem proving language meets some of your requirements. As an example:

``````structure Substring :=
( original : string )
( offset length : ℕ )
( invariant : offset + length ≤ original.length )
``````

In order to construct an instance of this `Substring` type, my code has to provide proof of that `invariant` proposition. Any function that consumes this type can rely on that invariant to be constrained by the compiler, and can also make use of that invariant to prove proofs about the function’s postcondition.

1. 0

I’ll drop this here again; your application configuration should be a relational database: https://lobste.rs/s/sr2lhm/pitfalls_using_env_for_configuration#c_k0d4l8

1. 4

This doesn’t engage with any of the issues outlined in the article.

1. 1

I don’t believe that any generic and general language can be used for configuration files

counterpoint: that language is SQL

the terms and elements of the domain are specific and custom to the program

this is relational data modeling

you cannot express this complex configuration directly in the language

correct, it is also expressed in the (relational) data model

Either you embed strings or data structures representing this logic in your general language

or you use a rich, relational data model with SQL for query and transformation

1. 1

How is configuration relational?

1. 2

In the same way that, for example, data for a customer invoicing system is relational. You define keys and relations for the domain entities, and attributes or reference relations for the details.

1. 2

I don’t think that business domain information is comparable to application configuration information. Application configuration isn’t a set of domain entities with relations to each other; it’s at most hierarchical/namespaced metadata, mapped to components in your dependency graph. Or? What am I missing?

1. 2

I’m not going to rise to the challenge from the article in order to completely re-model Apache vhost as relations, but to consider that example:

A virtual host is an entity in the domain of Apache configurations. It carries 1:1-arity relations to attributes such as `DocumentRoot` and `ServerName`. It defines a 1:many-arity relation to `ServerAlias` and a 1:zero-or-one-arity relation to `mod_proxy` attributes.

Deliberately modeling these data instead of using a text-based configuration can avoid caveats such as “IPv6 addresses must be specified in square brackets because the optional port number could not be determined otherwise.”

I totally agree that mapping the data to objects in the application dependency graph is a key requirement of configuration systems. If “ORM” is a dirty word to you, you may not like how I would propose addressing that. : )

1. 1

I guess my response would be that Apache “config” is extraordinarily complex, fails the test I describe here, and isn’t representative of the needs of the overwhelming majority of software.

1. 2

Quite frankly I’ve never needed to use xargs. For some of the examples here, using find with `-exec` is enough for me.

It’s quite a good idea for a program, and I can see where it would be useful, but I’ve just never needed to use it personally.

Edit: I just realised this exact thing was touched on in the article itself. xargs can be faster than find -exec.

1. 2

I was the “user asked why you would use find | xargs rather than find -exec” and the article ignores the rest of that comment thread where we pointed out that find supports batching the pathnames in that command and is not necessarily slower than xargs.

1. 1

The article mentions find -exec + and mentions the compositionality of xargs.

1. 1

Thanks. I figured there was more to it.
For the record, I’m still a die hard `find -exec` user.

1. 2

Teradata has a nice feature called “Query Bands” that allow the application to assign arbitrary key/value metadata to a session or transaction. We’ve used this to great success to keep track of which batch jobs were responsible for particular queries. An administrator can even use the workload manager to dynamically assign queries to different workload profiles.

1. 1

That is neat!

This sounds like a more structured solution like the query comments mentioned above by @viraptor.

From what I know: MySQL is also working on query attributes. Similar to connection attributes, but for queries. This comes very close to the Query Bands feature (without the workload manager part).

1. 42

Eh, there are some problems with xargs, but this isn’t a good critique. First off it proposes a a “solution” that doesn’t even handle spaces in filenames (much less say newlines):

``````rm \$(ls | grep foo)
``````

I prefer this as a practical solution (that handles every char except newlines in filenames):

``````ls | grep foo | xargs -d \$'\n' -- rm
``````

You can also pipe `find . -print0` to `xargs -0` if you want to handle newlines (untrusted data).

(Although then you have the problem that there’s no `grep -0`, which is why Oil has QSN. grep still works on QSN, and QSN can represent every string, even those with NULs!)

One nice thing about xargs is that you can preview the commands by adding ‘echo’ on the front:

``````ls | grep foo | xargs -d \$'\n' -- echo rm
``````

That will help get the tokenization right, so you don’t feed the wrong thing into the commands!

I never use xargs -L, and I sometimes use `xargs -I {}` for simple invocations. But even better than that is using xargs with the `\$0` Dispatch pattern, which I still need properly write about.

Basically instead of the mini language of `-I {}`, just use shell by recursively invoking shell functions. I use this all the time, e.g. all over Oil and elsewhere.

``````do_one() {
# It's more flexible to use a function with \$1 instead of -I {}
echo "Do something with \$1"
echo mv \$1 /tmp
}

do_all() {
# call the do_one function for each item.  Also add -P to make it parallel
cat tasks.txt | grep foo | xargs -n 1 -d \$'\n' -- \$0 do_one
}

"\$@"  # dispatch on \$0; or use 'runproc' in Oil
``````

Now run with

• `myscript.sh do_all`, or
• `my_script.sh do_one` to test out the “work” function (very handy! you need to make this work first)

This separates the problem nicely – make it work on one thing, and then figure out which things to run it on. When you combine them, they WILL work, unlike the “sed into bash” solution.

Reading up on what `xargs -L` does, I have avoided it because it’s a custom mini-language. It says that trailing blanks cause line continuations. Those sort of rules are silly to me.

I also avoid `-I {}` because it’s a custom mini-language.

IMO it’s better to just use the shell, and one of these three invocations:

• xargs – when you know your input is “words” like myhost otherhost
• xargs -d `\$'\n'` – when you want lines
• `xargs -0` – when you want to handle untrusted data (e.g. someone putting a newline in a filename)

Those 3 can be combined with `-n 1` or `-n 42`, and they will do the desired grouping. I’ve never needed anything more than that.

So yes xargs is weird, but I don’t agree with the author’s suggestions. `sed` piped into `bash` means that you’re manipulating bash code with sed, which is almost impossible to do correctly.

Instead I suggest combining xargs and shell, because xargs works with arguments and not strings. You can make that correct and reason about what it doesn’t handle (newlines, etc.)

(OK I guess this is a start of a blog post, I also gave a 5 minute presentation 3 years ago about this: http://www.oilshell.org/share/05-24-pres.html)

1. 10

pipe `find . -print0` to `xargs -0`

I use `find . -exec` very often for running a command on lots of files. Why would you choose to pipe into `xargs` instead?

1. 12

It can be much faster (depending on the use case). If you’re trying to `rm` 100,000 files, you can start one process instead of 100,000 processes! (the max number of args to a process on Linux is something like 131K as far as I remember).

It’s basically

``````rm one two three
``````

vs.

``````rm one
rm two
rm three
``````

Here’s a comparison showing that `find -exec` is slower:

Good question, I will add this to the hypothetical blog post! :)

1. 15

@andyc Wouldn’t the find `+` (rather than `;`) option solve this problem too?

1. 5

Oh yes, it does! I don’t tend to use it, since I use xargs for a bunch of other stuff too, but that will also work. Looks like busybox supports it to in addition to GNU (I would guess it’s in POSIX).

2. 11

the max number of args to a process on Linux is something like 131K as far as I remember

Time for the other really, really useful feature of xargs. ;)

``````\$ echo | xargs --show-limits
Your environment variables take up 2222 bytes
POSIX upper limit on argument length (this system): 2092882
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2090660
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
``````

It’s not a limit on the number of arguments, it’s a limit on the total size of environment variables + command-line arguments (+ some other data, see `getauxval(3)` on a Linux machine for details). Apparently Linux defaults to a quarter of the available stack allocated for new processes, but it also has a hard limit of 128KiB on the size of each individual argument (`MAX_ARG_STRLEN`). There’s also `MAX_ARG_STRINGS` which limits the number of arguments, but it’s set to 2³¹-1, so you’ll hit the ~2MiB limit first.

Needless to say, a lot of these numbers are much smaller on other POSIX systems, like BSDs or macOS.

3. 1

`find . -exec blah` will fork a process for each file, while `find . | xargs blah` will fork a process per X files (where X is the system wide argument length limit). The later could run quite a bit faster. I will typically do `find . -name '*.h' | xargs grep SOME_OBSCURE_DEFINE` and depending upon the repo, that might only expand to one grep.

1. 5

As @jonahx mentions, there is an option for that in `find` too:

``````     -exec utility [argument ...] {} +
Same as -exec, except that ``{}'' is replaced with as many pathnames as possible for each invocation of utility.  This
behaviour is similar to that of xargs(1).
``````
1. 4

That is the real beauty of xargs. I didn’t know about using + with find, and while that’s quite useful, remembering it means I need to remember something that only works with find. In contrast, xargs works with anything they can supply a newline-delimited list of filenames as input.

1. 3

Yes, this. Even though the original post complains about too many features in `xargs`, `find` is truly the worst with a million options.

4. 7

This comment was a great article in itself.

Conceptually, I think of xargs primarily as a wrapper that enables tools that don’t support stdin to support stdin. Is this a good way to think about it?

1. 9

Yes I’d think of it as an “adapter” between text streams (stdin) and argv arrays. Both of those are essential parts of shell and you need ways to move back and forth. To move the other way you can simply use `echo` (or `write -- @ARGV` in Oil).

Another way I think of it is to replace `xargs` with the word “each” mentally, as in Ruby, Rust, and some common JS idioms.

You’re basically separating iteration from the logic of what to do on each thing. It’s a special case of a loop.

In a loop, the current iteration can depend on the previous iteration, and sometimes you need that. But in `xargs`, every iteration is independent, which is good because you can add `xargs -P` to automatically parallelize it! You can’t do that with a regular loop.

I would like Oil to grow an `each` builtin that is a cleaned up xargs, following the guidelines I enumerated.

I’ve been wondering if it should be named `each` and `every`?

• `each` – like xargs -n 1, and `find -exec foo \;` – call a process on each argument
• `every` – like `xargs`, and `find -exec foo `+` – call the minimal number of processes, but exhaust all arguments

So something like

``````proc myproc { echo \$1 }   # passed one arg
find . | each -- myproc  # call a proc/shell function on each file, newlines are the default

proc otherproc { echo @ARGV }  # passed many args
find . | every -- otherproc  # call the minimal number of processes
``````

If anyone has feedback I’m interested. Or wants to implement it :)

Probably should add this to the blog post: Why use `xargs` instead of a loop?

1. It’s easier to preview what you’re doing by sticking `echo` on the beginning of the command. You’re decomposing the logic of which things to iterate on, and what work to do.
2. When the work is independent, you can parallelize with `xargs -P`
3. You can filter the work with `grep`. Instead of `find | xargs`, do `find | grep | xargs`. This composes very nicely
1. 9

My favorite PAGNI arises when building distributed systems with message queues like Kafka or JMS is having an error or “deadletter” topic. Inevitably some messages will get into the system that cannot be processed for some reason, and you need to have a way to get them out of the main channel so they can be handled separately by a manual operator or a special consumer.

1. 14

I worked for a while in a comp sci lab that specialized in solving real-world problems with statistical analysis and (generally non-neural-network) machine learning. The PI was quite good at making contacts with interesting people who had interesting data sets. People would come to him with “big data” problems and he would say “this fits on a single hard disk, this isn’t big data. We can work with your data far more easily than these other people with Hadoop pipelines and whatever, you should work with us.”

We had a compute server with 1 TB of memory, in 2016, and it was not particularly new. Turns out that if you’re counting things that exist in the real world, there’s not that many actual things that require a terabyte of RAM to keep track of. It probably cost low six figures to buy, or rather, about the same as one full-time engineer for a year. (Or two grad students, including tuition.)

I didn’t do particularly well at that job, but I did learn that 90% of the work of any data-oriented project is getting your data cleaned up and in the right shape/format to be ingested by the code that does the actual analysis. `xsv`, `numpy` and similar tools can make the difference between spending a day on it and spending a week on it. That was far more fun for me than the actual analysis was.

1. 15

By contrast, at a previous gig I watched an entire data science group basically fuck around with Amazon and Google Bigtable and data lakes and lambdas and Kafka to handle ingests of a whopping…three events a day? maybe?

Their primary deliverable every month seemed to be five-figure AWS bills occasionally punctuated with presentations with very impressive diagrams of pipelines that didn’t do anything…my junior I was working with, by contrast, was whipping together SQL reports and making our product managers happy–a practice we had to hide from the org because it was Through The Wrong Channels.

And because reasons, it was politically infeasible to call bullshit on it.

1. 2

Sounds like your last gig was wasting their time and money really.

1. 1

I hope you didn’t spend too long there. It sounds hellish

2. 5

Really understanding what problem you’re actually trying to solve is often overlooked in the desire to jump on the latest buzzwordy technologies.

Every time BitCoin power consumption comes up, I go and look at the transactions per second that the entire BitCoin network has done over the last week. I’ve never seen it average more than 7/second. If you want to use a cryptocurrency as a currency (rather than the exciting smart-contract things that things Etherium allow, which may require a bit of compute on each transaction), each one is simply atomically subtracting a number from one entry in a key-value store and adding it to another.

A Raspberry Pi, with a 7 W PSU, could quite happily handle a few orders of magnitude more transactions per second with Postgres or whatever. Three of them in different locations could manage this with a high degree of reliability. You could probably implement a centralised system that outperformed Bitcoin with a power budget of under 50W. BitCoin currently is consuming around 6 GW. That’s a roughly 8 orders of magnitude more power consumption in exchange for the decentralisation.

1. 4

Really understanding what problem you’re actually trying to solve is often overlooked in the desire to jump on the latest buzzwordy technologies.

Perversely, the problem that’s often being solved is “keeping the engineers from getting bored”, “padding your resume to make it easier to jump”, or “making your company more sound more important to attract VC dollars”.

1. 3

Now that I think about it, “If we use technology \$x we’ve got a better chance of nabbing VC money” can often be a sound business decision.

2. 2

Every time BitCoin power consumption comes up, I go and look at the transactions per second that the entire BitCoin network has done over the last week. I’ve never seen it average more than 7/second. If you want to use a cryptocurrency as a currency (rather than the exciting smart-contract things that things Etherium allow, which may require a bit of compute on each transaction), each one is simply atomically subtracting a number from one entry in a key-value store and adding it to another.

This is part of the design. Every 2000 blocks or so the protocol adjusts the difficulty of mining to keep the average mining rate at roughly 1 per 10 minutes.

1. 1

more like 3 microwatts if you’re being super sloppy

2. 4

It probably cost low six figures

IIRC, my infrastructure team told me the cost to replace our 1U 40-core IBM server with 1TB RAM was going to be around \$50k

1. 4

Just checked. 48 core AMD EPYC cpu, 1TB RAM, no disks past boot, 1U: just over \$18k. Call it 20k with a 40G ethernic and a couple TB of NVMe.

1. 1

That’s a pretty affordable kit! The 1TB of RAM (8x128GB) alone from SuperMicro or NewEgg would cost \$10k-\$15k.

1. 3

That’s listed price from a SuperMicro VAR, checked this morning.

1. 2

I was talking with my team again today and this came up, that \$50k price tag was actually for a pair of servers.

2. 1

Turns out that if you’re counting things that exist in the real world, there’s not that many actual things that require a terabyte of RAM to keep track of.

I’ve been saying this for years. I haven’t used numpy too much—I’m not a real data analyst—but I’ve gotten the job done with SQLite, and/or `split; xargs -P | mawk`.

1. 23

This is a brief memoir (by Bob Martin) which offers almost nothing in terms of type theory, and only 4 paragraphs are relevant to Clojure

1. 110

Independently of what burntsushi decides, I think that the following concrete actions would be in order:

1. Remove the last half-sentence of the banner that explains how to delete your account. Having it as the last part of the message adds a lot of weight ot it: after you read the banner it’s the main thing you remember, the oddly detailed explanation of how to erase yourself.

2. Reformulate the rest of the banner to indicate that this is a heuristic that may be wrong. It’s important when providing hints from data to tell people explicitly that we know the hints may be wrong. Otherwise it feels very different.

Let’s rephrase this banner with the explicit goal of doing no harm (better, doing good) when it is wrongly displayed, instead of thinking about the formulation when its assessment is correct.

1. 59

Speaking as somebody who’s gotten that banner basically permanently on their account, I think a lot of folks here may be missing important context:

• The call to delete your account makes a lot of sense if you have to handle users dramatically declaring, in DMs/IRC/elsewhere “One more and [clutches pearls] I shall leave, forthwith!”. This is tiresome from a moderation standpoint and annoying for a community, and if one can nudge somebody into removing their account–in the common case!–rather than continue to engage in a community they don’t fit into (for whatever reason), it’s a win for both parties.
• This isn’t some randy sketchy heuristic–the warning links directly to a histogram that shows where you are in the flaggings. It’s deterministic, it’s impartial, and if you’re getting flagged a lot there’s something going on with either your posting or the community.
• Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.
• One of the biggest reasons for Lobsters coming from the orange site was @jcs being annoyed at opaque moderation. While I might disagree certain particular actions, there is a public log. The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.
• There’s a trend for folks to go on IRC, Twitter, or wherever and talk trash about our community, about how @pushcx is powertripping, or whatever. This behavior probably plays well with the circles they’re familiar with and probably feels like punching up, but it’s gotta suck for the person who manages the community to have people he’s tried to work with and accomodate throw it back in his face.
• People are really flag-happy, and this is what happens when you are.

(Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

1. 35

Moderation team here is currently basically just @pushcx. It’s a large community with real, actual bad actors, and asking for any sort of increase in manual moderation is a very real burden.

This bit deserves signal boosting IMO.

Lobsters doesn’t cost anything to join or participate in. The people who run it are clearly doing it for love, not money.

Speaking for myself I’d much rather get past the momentary ouch of having a red warning message attached to my account than have the owners or moderators rage quit as a result of feeling UTTERLY drained by the experience.

I’m watching the life get sucked out of some very stalwart well meaning people in another community I care about simply due to the sheer VOLUME of constant negative feedback, so I think we all owe it to the mods to suck it up and cut him/them some serious amounts of slack.

1. 22

<3 Thank you very much.

2. 24

(Minor annoyance: we used to have a very good explanation of how to use flags, what borderline cases looked like, and so forth, but that was purged for whatever reason. I’ve seen a lot of bad flagging by either new folks who don’t know better or users with an axe to grind.)

I would love to see this come back in some form. As someone who joined after that was removed, I feel there isn’t enough information about flags. Every single time I’ve ever flagged something, I’ve been really hesitant because I wasn’t able to find any guidance on what qualifies to be flagged.

1. 3

I agree. I actually don’t think this is so minor on its own though I can see why you phrased it that way in its original context.

2. 23

I realize I’m not as active as I might be - I’m bipolar, my ability to spend time on things comes and goes - but I promise that I do still pay active attention to the site and step in when it seems warranted. Also, the mod team has lots of discussions behind the scenes. Just because @pushcx is the face of a decision doesn’t mean the rest of us aren’t involved.

I felt I should address that, since you mentioned it.

Edit to add: Thank you for the kind words, overall. It’s helpful to know that you and others do see the point of the banner.

1. 22

People are really flag-happy, and this is what happens when you are.

Speaking from experience as someone who’s moderated some decent-sized subreddits – where reddit has both the ability to downvote, and a separate ability to “report” for posts/comments that break subreddit or site-wide rules – this is absolutely the case. The number of people who just spam-click the “report” button thinking it’s some sort of super-downvote that will remove the thing they dislike is astounding. Same with people who periodically decide they just don’t like some type of content even if the subreddit generally allows it, and go mass-report a hundred posts to express their displeasure.

1. 11

Yeah, this is what I think is fundamentally the issue. A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button. Perhaps in their mind, if the post was deleted, then the opinion no longer exists or is somehow invalid???

I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

People simply think people who disagree with them are either stupid, or trolling (deliberately saying stupid things hoping for a reaction) or both.

1. 12

A lot of people simply have no emotional control. They have strong believes but no ability to defend them. With that, when they see opinions they dislike, instead of engaging with a retort, they will use the report button.

It’s also true that people sometimes repeatedly post tired nonsense, and that merely (and at obviously low cost) repeating something that’s poorly researched, or broadly offensive, doesn’t entitle the poster to a vigorous debate. Sometimes the use of a flag is the only viable option.

1. 4

If somebody posted some tired nonsense then it should be very easy to dispute.

doesn’t entitle the poster to a vigorous debate

You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

Sometimes the use of a flag is the only viable option.

Anybody could deem all your posts ‘tired nonsense’ and just report all of them? That is as legitimate as when you do it. Do you somehow think that your idea of what is tired nonsense is universal and therefore a good metric for when a post should be reported?

1. 8

If somebody posted some tired nonsense then it should be very easy to dispute.

In the case of trolling, the whole point is to trick people into taking the bait, which I believe moves the subthread higher up in the scoring ranks.

You don’t have to reply. The issue here isn’t whether or not to reply but whether or not reporting is an appropriate response to somebody saying something that you think is wrong and the answer is no.

I think that detecting trolling is a skill that’s pretty key to the survival of a community, but a good troll will make it quite difficult to resist engaging. Trolling isn’t commonly overt, over the top rudeness. Way more often it is an attempt at looping other community members into a “debate” that ends up in people pointing edgy and charged hot takes at each other.

In its worst form, it will promote socially harmful conclusions through pseudo-from-first-principles lines of reasoning that are aesthetically attractive to the type of people that visit tech forums. To dispute requires dissecting phony arguments, which at a glance, appears to legitimize the “debate” and grant the troll a certain level of community approval, especially if they’re good at giving off the appearance of rationalism. IMO ignoring and flagging this type of content is nowhere near what’s required fully address it.

1. 3

How are you not using an accusation of trolling as a free ‘You lose’?

How do you know that something is true? It is via putting your ideas up for others to engage and debate. But you want to simply circumvent that by calling opinions you don’t like trolls, and then saying that those opinions don’t even need to be debated or engaged because that will legitimise it.

Sooner or later instead of having attacking each other’s ideas so we can improve then, all we will end up doing is claiming the other side is ‘trolling’ so they are immediately wrong and don’t even need to be disproven.

Why even bother defending your ideology against opposition, when you can simply claim the opposition is a foreign spy or a mentally ill person and then getting rid of them?

2. 5

I have gotten the banner too. I thought the words were harsh. But over the years I have noticed that some of my posts will follow the pattern of (+10 upvote, -8 troll), except the troll downvote count has increased over time. Based on this I assume that the general ratio of people on this site who like to report opinions they don’t like as troll versus people who are willing to engage in intelligent discussion has increased.

Or maybe you’ve gotten more and more trollish over time.

1. 5

Or maybe we have too many politically driven ideologues who are not interested in communication and they simply throw the word troll around to avoid having to engage with their opposition on the idea plane.

1. 9

Nah, I’ve been around just about as long as you have, I can count on one hand the number of times I’ve not seen you be a troll in a thread. I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread. Pretty much all you do is get into fights around here.

1. 3

Devil’s Advocate: Perhaps @LibertarianLlama is among a new breed of Lobsters who are also concerned with the human side of things, not just the technical one.

1. 2

I’ve not seen you be a troll in a thread

I suppose you are one of those people who have a very low margin for what constitutes ‘trolling’.

I don’t even know what technical subjects you have expertise in, since I’ve never seen you contribute technical insight to a thread.

I didn’t realise technical expertise in a subject is a requirement of using this website. Regardless you can just assume that I have no expertise in any subject.

2. 6

I wonder if a literal “super downvote” button would work as a psychological trick. It would act as a normal downvote, but just look like a bigger, angrier button that would absorb more anger from whoever clicks it. (At the same time, possibly rate-limit the actual report button per user…)

1. 8

That’s a pretty cute idea. I’m tempted to say that, for people who have the self awareness to realize that what they mostly need is catharsis, I recommend getting it from their favorite action videogame, or a trip to the gym or whatever… I don’t want to dismiss your idea though, it’s true that it could help to have some sort of reminder or prompt to seek catharsis.

1. 3

Edit: someone has looked at the system and much of this is actually already automated(!): https://lobste.rs/s/zp4ofg/lobster_burntsushi_has_left_site#c_qiipbc

Previous:

OK, another cute - or hopefully even better - useful idea:

I think HN has a pr account setting that ignores flag from that user, I’m fairly certain dang has mentioned at some time that one should be careful with flags. (Now that I think about it it might have been the vouch option he was writing about.)

I’m not really sure how it would work with perfectly transparent moderation:

• would the user get a warning?
• or would it just show up in the moderation logs?
• maybe it is a fraction so if a user has 1/5 flag weight it takes more than 5 such users to flag before it counts as one ordinary user? Then the moderation log could still say “flagged based on user feedback” or whatever it say but omit the details about there being 10 flag abusers counting as two full votes and 3 ordinary users?
1. 11

Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

1. 1

Good thoughts. On Lobsters, if I know that a user is going to continue using flags vindictively, I would rather just ban them… of course, I’d talk to them first.

Automation in this sort of thing can be a false savings because if we set up a certain automated rule, we are implicitly sending the message that any behavior that falls within the rule is allowed. So in your example, I would expect to see users who intentionally stay just below the threshold for having their flags de-valued, as well as users who intentionally go past the threshold and flag in a wanton manner because they know that the punishment for doing that is just that their flags will have less numeric weight. Having it be a socially enforced rule, counterintuitively, can often lead to better behavior overall compared to having a technical enforcement mechanism.

2. 5

The moderators of r/TheMotte (a subreddit which has the best moderation IMO) observed pretty much the same:

No one who has seen a mod queue would be surprised by this. Even those of you who are always very calm and nonconfrontational, with middle-of-the-road opinions, might be shocked how often your posts are reported. Some of the most reasonable, unspicy posts get marked as “It’s targeted harassment.” Some people will report “Inflammatory claim without evidence” just because they don’t agree with the post.

I pretty much started brushing aside (like several other users reportedly do) the red warning that lobste.rs throws in your profile for this reason, as well as for that condescending final phrase “Reconsider your behaviour or take a break” which rests on the erroneous assumption of there incontrovertibly being sound rationality in this community’s flagging behaviour.

I have a theory that this became worse after the site removed the downvote functionality thus removing a potential anger-outlet for users, who have now shifted to channel that affective energy via flagging. This theory however can only confirmed by doing a simple analysis of the data (was there an increase in flagging right after disabling downvoting?)

3. 12

It’s irrelevant that it is impartial if it is exploitable. Which it very much is.

So the system allows for a mob to harass an user into leaving the site, but you claim that it is important to note that anyone can be harassed. That is the matter subject to challenge here, not “important context”.

Yes people are trigger happy, for that reason, the flagging feature might be counter productive.

1. 5

you claim that it is important to note that anyone can be harassed.

I’m not sure I see where you got that from my post.

The ability of the mob to harass users here exists, and is exploited, at scale and quite apart from this notice that “yo, you’re getting a significant number of flags.”

It doesn’t make sense to optimize for people who run away when told others disagree with them.

1. 21

No one wants to be told they don’t belong here. I really enjoy Burntsushi and a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

1. 6

OK I see an important distinction here. I tried to point this out to burntsushi on Twitter.

If you read the message he got VERY carefully, it sets forth a very specific sequence of potential and suggested routes to mitigation. They are:

1. Take a breath. Take a break. Step away for a bit and think about whether maybe the problem might exist at least partly in your head and the resultant behavior you’re exhibiting in the community.

2. Talk to a moderator about the situation.

3. Then, and ONLY then, does the message point out that you can if you so choose also delete your account.

Having gotten the big red warning myself a ways back, I DO sympathize that it’s an ouchy thing to have happen in the moment, but I strenuously believe that the intent here isn’t to make anyone feel unwelcome. Just the opposite, my read of the intent is an earnest attempt on the part of the community to help people moderate their behavior, and if that’s not possible or applicable, to seek other options for mitigation.

1. 11

No one wants to be told they don’t belong here.

I’d agree, but we have done very poorly by some of our users in that regard already and the mob likes it that way. Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff? Peppridge farm remembers.

There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe–and they’ll get even angrier if you suggest that maybe, possibly, we should focus on something else.

So, while I agree in principle with you, the community has through its actions demonstrated that it does not care.

Your response of “I don’t want people with a thin skin”, is itself thin skinned in the opposite direction.

I don’t believe that it is. You are, of course, welcome to your own interpretation.

1. 14

Remember that time recently the mods banned one of (if not the) current expert on evidence-based software engineering because of an off-hand comment touching culture war stuff?

IIRC he had a pattern of problematic behavior, and also I’m not sure how much of an expert in ESE he actually is: his book is overwhelming, but keeps giving me bad vibes. I’m struggling to explain what it actually is, but I’m really suspicious that there are serious problems with it.

1. 1

I’d definitely be interested to hear more.

I’m struggling to explain what it actually is

Mind giving it a try? I couldn’t see anything particularly bad (relatively speaking) in their comment history, nor anything related to the ban rationale. Seems like you’ve been paying more attention, haha

2. 14

There are no shortage of Lobsters who will clack and freak out if made aware of anybody who doesn’t appear to be in their political tribe

The basic problem with statements like this is that nearly everyone can feel it applies to the angry mob that’s disagreed with them.

Personally I’m on record as believing that there is no way to separate “politics” from what we do (simple example 1, simple example 2) or to write software without considering the impact it will have on people.

But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue. The hypothetical guy (example 2 above) with no criminal record who can’t get bail because the system could only store his name as a bunch of “aliases” is “politics” to me, but apparently not to many other people (it’s just “falsehoods programmers believe about names”), while holding a software project’s leader accountable for misbehavior is “politics” to many of those people and common sense to me. A staggering percentage of actual “political” fights really are that sort of thing.

1. 3

But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted, which says to me that it’s not that people don’t want “politics”, it’s that their definition of what is and is not “politics” is at issue.

I mean the proof is at hand right? A “let’s not discuss politics” non sequitur as successful troll.

1. 0

But I also see plenty of “oh woe is me, the cancel culture and the woke and the leftist and the SJW are taking over” stuff that gets left in place and even highly upvoted

I don’t think I’ve ever seen this on Lobsters, only on others sites. Do you have any examples?

2. 1

I think we disagree about what it means to be thin skinned. I thought you were admonishing the people who leave into toughening up. If that is the case, I think it will lead to either an echo chamber or a dungeon of trolls or both.

But based on your other comments, I don’t think you meant what I interpreted.

You in this context means, us, or the offended user that flags people into quitting. How about we make flags public and you can only spend them in certain ways? Do comment points even mean anything? There is a huge distinction between “this person is harmful” and basically anything else. if someone is harmful, they should be removed. Otherwise … if you don’t want to see their posts, then that would be a personal choice.

1. 2

Making flags public is a terrible idea.

1. 2

I agree. I DO think that people here are overly trigger happy with the flagging, but given other comments in this thread I’m willing to believe this is simply human nature at plan and unavoidable. Making flags public would be in effect shaming someone for trying to make a small contribution to the community by flagging a post.

1. 2

I feel that maybe 60% of flags are not trying to contribute to the community, but rather to avenge themselves, or to have an outlet for disagreement without having to engage. It’s part of the problem with the Internet at large: for digital natives, disengagement is no longer an option. People can’t just ignore stuff they disagree with; they have to respond to it, and if it’s a troll, they shouldn’t engage directly, so they instead engage indirectly by flagging. There’s a reason why Twitter’s first option when reporting is “I disagree with this”. It’s just human nature.

2. 1

Because? Without justification your statement is opinion, what am I to do with that?

1. 1

Because it can lead to retaliatory, tit-for-tat flagging of the flagger’s comments. Mods can already see who flags what and can take action against those who misuse it.

2. 2

a system that is constructed that make it so people like that dont want to be here is a system that needs to be fixed.

Who are “people like that”? To me, this sentence seems to extrapolate based on a single data point, which is based on some observations of another person. That is to say, we don’t know Burntsushi well enough, and maybe no matter how we setup the system, he would find something to quit. And striving to optimize for one would put burden and alienate others.

1. 5

people like that

Hardworking, smart, helpful, writes a lot of great Rust. Reddit and github

I am talking about specifics not a generalized idea of a Lobste.rs user.

2. 0

Since that person left, I’m not getting a flag on every single of my comments anymore.

An interesting coincidence. ¯\_(ツ)_/¯

1. 0

Obligatory “correlation is not causation”.

3. 5

That is the problem right there. It.s not a “disagree” link, it’s a “flag” link. Obviously, looks like there are people abusing it because they disagree with something rather than unacceptable behaviour being reported.

Personally, I wouldn’t take it too seriously much less posting on other websites about it. But it is rather shitty to be told to take a chill pill just because one’s opinion is not widely accepted.

1. 1

Accounts with low “karma” can’t downvote but can flag. Perhaps this needs to be looked at..

1. 6

There are no downvotes, only flags.

1. 2

TIL. I stand corrected

1. 1

Thanks for clarifying this. I was wondering where the downvote links were.

1. 4

iirc the downvotes were the flags, just with a downvote arrow instead of being hidden.

2. 3

It doesn’t make sense to optimize for people who run away when told others disagree with them.

I think discourse is improved when all participants feel safe. Then instead of spending energy in handling negativity emotionally, they can make better arguments. Moreover I don’t want to see valid viewpoints shut down due to mob shouting. I think human discourse works well with a variety of opinions.

1. 16

I think human discourse works well with a variety of opinions.

I agree!

I think discourse is improved when all participants feel safe.

The problem we’re seeing in the last several years is that there are people who will never feel safe, at least not without massive sacrifices from everybody else in a community, often including systematic self-censorship and topic avoidance. The great online community question of the age is how to strike a balance between being welcoming and letting your demographics become taken over by–for lack of a better way of putting it–people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem.

1. 9

people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

1. 15

An (only slightly contrived) example:

You run a message board that hosts tech content.

On one extreme, you tell everybody “hey, we want you here! everybody is super nice and you’ll never be criticized!”.

After some period of time, you end up with a bunch of users of C and a bunch of users of Rust.

Some of those users cannot bear any perceived slight of their chosen language. There are C people that go off in a tizzy if anybody points out the memory safety issues in C, and there are Rust users that freak out if anybody complains about long compile times. The house rules–because remember, we’re optimizing for welcoming and that means supporting everybody, no matter how thin-skinned they are!–end up including “Never talk about shortcomings of another language.”

Since engineering is all about tradeoffs, and since no language is actually good for all things, strict adherence to this rule ends up with only the most optimistic and vapid discussion–worse, it infects other topics. It’s really hard to explain why you might like immutability for concurrency if you can’t talk about memory safety issues because you’ll trigger the C programmers. It’s really hard to explain why you like dynamic or interpreted languages if you can’t talk about how much long compile times suck because you’ll trigger the Rust programmers.

On the other extreme, you tell everyone “hey, we talk about tech and things on their own merits here, and if you can’t stand commentary that offends, fuck off!”.

Our C users and Rust users come in.

Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

Obviously, there will be users who would be knowledgeable assets to this community but who are squicked out by this stuff and leave. There are other users who can put up with the rampant nonsense but who don’t want to spend the cycles required to weed through the bullshit to talk about what they want. Over time, the community stagnates unless it can find a way of attracting and retaining new users, and odds are that’ll be based on its reputation for no-holds-barred “discussion” rather than actual technical insight…this userbase will also skew towards younger folks who have a deficit of experience and all of the time in the world to share it. This is, incidentally, how /g/ works.

~

I’m not sure what kind of online future you’re envisioning but I don’t want any part of it.

My point isn’t to advocate for one or the other, but to point out that it’s a central question of community.

It’s also a question whose answer shifts over time, as groups tweak the trajectory of their tribe.

1. 8

Some of these users are just assholes. There are C programmers that strictly use bizarre sexual analogies for anything involving pointers, there are Rust coders that sign all of their posts with Holocaust denial, frequently threads devolve into seeing who can outlast who in massive shit-slinging, and in general a sport is made of making miserable and hazing anybody who won’t stick around.

And there’s a pretty easy way to fix that – kick ’em out. There are communities which do that.

Because as the saying goes, there are two ways to be a “10x programmer” and one of them is to have such an awful effect on people around you that they drop to 0.1x their prior productivity while you stay average. And even if someone really is “10x” (whatever that means) on some aspect of technical skill or knowledge, it’s still likely to be a net loss to put up with “asshole” behavior from them.

1. 2

This is good advice and knowledge for me as I try to build one or more Open Source communities. I think you are correct that such communities should have no room for such users.

1. 1

For the purposes of that thought experiment, we take the position in the second case where we do not eject members for being assholes.

2. 3

I saw another attempt at creating a HN clone go down a few years ago, not because of flaming or lack of interesting users but because some early users decided they and everyone else decided they needed to be much nicer than HN.

I quit commenting soon after.

You can check my comment history, I’m not at rude person but those rules made me feel unsafe. I started carefully considering each word. Then stopped posting,

I wasn’t the only one. The “super nice” crowd didn’t have anything else to contribute and the site was down a few months later.

1. 8

Honestly, one of my main issues with HN has always been the focus on superficial civility. You can be an absolute horrendous awful drive-others-away-in-droves trainwreck of a poster there so long as you do it the “right” way. And you will be valued and rewarded for it.

A real “anti-HN” would care less about those superficial aspects and more about dealing with the actual trolls and assholes.

3. 4

people who are not robust enough to survive or flourish in a sufficiently diverse/hostile memetic ecosystem

Perhaps I don’t see this as an issue. I believe, in my ideal community (where I’m stressing “ideal” because it’s probably not practically realizable), that any commenter with a good-faith argument should feel empowered to make that comment, even if others in the community viscerally disagree. There are two problems at hand here, one is defining what “good-faith” mean and the other is protecting arguments from being shouted out. When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements. In my mind, the easiest way to come to a consensus on what “good-faith” means is to clearly define it. This is why I’m a fan of Code of Conducts. They lay bare what a community considers “good-faith”, so the minimum bar of participation is met. Ideally the CoC would change infrequently and have community buy-in so that it acts as a minimum bar to entry.

Ideally (again the ideal here not the practice, probably), we can create a community where there is no hostility after agreeing to “good-faith” rules. It’s one thing to enter heated discussion, but I’m hopeful that a community can use its own moderation tools to ensure that a heated discussion does not turn hostile.

I just don’t see a world where diversity and hostility are equivalent.

1. 1

When it comes to “good-faith”, due to the nature of human ideology, there will be disagreements.

I make good-faith arguments all the time that seem to fall into the class of bad-faith arguments that @bitrot points out, so I get what you’re saying.

I just don’t see a world where diversity and hostility are equivalent.

In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄

At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

1. 2

In the interest of naming things, it seems what’s under discussion is the paradox of tolerance. I’d link to Wikipedia, but it seems it’s a Marxist idea, which I know not everyone would be comfortable with 😄

Indeed, I am familiar with the Paradox of Tolerance. My hope is that a CoC or otherwise similar guiding document or constitution would provide a floor on tolerance, but everything beyond that would be tolerated. Not only does that set community baseline rules but it keeps rules distinct, clear, and constant. There’s a reason many real, representative governments go through due process to change laws.

At some point, I think you just have to go by how things seem. Which is a shame, because that makes it a game of “survival of the fittest”, where those who are best at branding, or putting things “the right way” to seem good to other people, survive; while those who are prone to misrepresenting themselves, or discussing “untouchable subjects” such as the example in @friendlysock’s comment, are booted out.

Perhaps the conclusion I should arrive to is that direct democratic (e.g. “upvote” and “downvote”) discussion sites just lead to dynamics that create and sustain echo chambers. I’m not sure, it’s certainly something I’m mulling about myself.

2. 2

when told others disagree with them.

But that’s not what flagging tells you… If you’ve been doing everything alright, it tells you that others who disagree with you have leveraged a system unrelated to disagreement to make you look unreasonable, unkind, and/or trollish.

3. 4

I’d be interested in learning if mods have any way to detect brigading, possibly such as by looking at correlations in users who consistently flag the same comments.

1. 13

Yes, we do. If you look at the mod log you will see that people are banned for this practice now and then. Don’t do it.

1. 1

I’m not sure why you needed to admonish me like that at the end. I wasn’t probing your defenses. I was merely going to suggest this if you didn’t have it.

No need to reply, I know this is a tough time for all the staff.

1. 7

Sorry, I should have been clearer that I wasn’t directing that at you in particular. I know you were trying to help.

4. 8

Preface: I don’t have his comments handy and am just looking at the system. Despite this I will be embarrassed if it turns out he was actually making an enormous mess 🤣

Also want to add whoever worked on the existing system clearly put a lot of thought into it (e.g. factoring in flags on multiple stories, percent-flagged, and limiting to N users all seem wise, and thought went into the message). Probably no fun to see someone armchair quarterbacking with much less time thinking about the problem space than they have. Hope this at least crosses the threshold of possibly useful.

The thing here being complained about? It’s in the source code, it’s very clear how it works, and it provides impartial, statistical feedback about what’s going on.

It looks like this is it. My guess is if you’re prolific, it’s inevitable you’ll pop over the hard numerical thresholds sometimes and the 10% of comments is the biggest barrier, followed by the top-30 ranking.

10% of comments seems not that unlikely if 1) your recent comments are seen by lots of people (upvoted, early, etc.), 2) they address things a lot of people disagree about, and 3) some % of folks who disagree will flag, which seems likely as long as flagging is one of the easier ways to express disagreement (more below). You could do something like subtract the upvotes/N from the flag count (lots of fiddly options), just because there isn’t a number that represents visibility directly.

The top-30 ranking has has the same considerations as flag count, and also if discussion is going well enough, the 30th “worst” person might not be that bad. (Somebody’s the least polite person in a group of absolute diplomats.) The histogram/rank data does seem interesting to expose to the more flagged folks. Given all that, I’m not entirely sure what I think about the wording of the message–definitely already phrased to aim for “stop and consider this” not “you’re a bad actor”, but I can also see how humans get defensive.

People are really flag-happy, and this is what happens when you are.

(Added clarifications here in [brackets] after friendlysock’s comment.)

A nice thing about [the idea of downvotes for simple disagreement, as used on some other sites, is that it lets] people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on. I think the dream behind the current system is that not having [no-reason] downvotes leads to a very pluralist site where many people disagreeing (or fewer disagreeing strongly) doesn’t count against a comment as long as there’s no objective problem with the phrasing/on-topicness/etc. Realistically it seems as if without another outlet, some fraction of disagreement will get funneled into the flag system (even just through people looking hard for technically valid reasons to flag) even if you try to discourage it.

Just as a random thought, you can offer downvotes without treating them as -1 upvote (-0.5 or -0.25 or whatever in a ranking algorithm), or disagree flags that you treat differently from others.

tl;dr if I guessed at possible tweaks they might be allowing [no-reason] downvotes or disagree flags that have less impact than existing flags, just to keep that self-expression out of the flagging system; look for a way to factor in that being more-read will get your comments more negative attention; and maybe there’s something to do with the message wording, though it’s clearly already had a good amount of thought put in it.

1. 16

A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

Yes, I very much agree with this. On Reddit and Hacker News, which lobster.s is modelled after, I think of as:

• upvote: “I would like this to appear higher on the page”
• downvote: “I would like this to appear lower on the page”

Flag is a totally different thing than downvote – it means that the user is abusing the site, and continuing that behavior SHOULD result in getting banned from the site.

But expressing minority opinions (thoughtfully) may result in downvotes (*), but it should NOT result in getting banned or encouraged to leave.

As far as I can tell, the burntsushi warning was a result of mixing up these signals. I only read a portion of the Rust threads, so I haven’t followed everything, but from what I understand he was probably just expressing some opinions that people disagreed with, not actually abusing the site.

So I think you have diagnosed it correctly – the lobste.rs UI is sort of mixing these 2 signals up by omitting downvotes. The “flag” button has 5 reasons, and “off topic” and “me too” should be a downvote, not a flag IMO.

Again, the difference between the two is whether continuing the behavior should lead to a ban, and that’s a very important distinction.

(Honestly I am surprised that I haven’t gotten this banner ever, given that there is a minority of people who disagree with my opinions on shell. Since I’m writing a new one, I’m opinionated about it, but also invite disagreement)

(*) Some people may want opinions they disagree with to appear lower on the page, and some people might not. In my view it’s up to person to exercise that discretion – that’s why you get one vote :)

1. 3

upvote: “I would like this to appear higher on the page” downvote: “I would like this to appear lower on the page”

That’s not actually what they mean on Reddit:

Vote. If you think something contributes to conversation, upvote it. If you think it does not contribute to the subreddit it is posted in or is off-topic in a particular community, downvote it.

‘I would like this to appear higher/lower on the page’ is so ambiguous that it leads to different people misunderstanding it and applying it differently.

1. 6

I’d say that’s a prescriptive definition, but a descriptive one is more useful [1], and thinking about it that way will be more likely to solve the problem.

That is, “users don’t read documentation”, and it means whatever the users think it means, and it changes over time. FWIW I’ve been using Reddit since 2007, and HN since 2011 or so, and there’s never complete agreement on what these things mean.

I’ve seen the debate about downvote vs. disagree several times over the years on the sites. I choose to sidestep it with the vague definition – that’s a feature! :) It’s OK to be vague sometimes.

In cases like this, where the number of flags leads to either banning or encouragement ot leave, I think it’s better not to be vague, and have 2 distinct mechanisms. This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

1. 2

This isn’t a big change, as lobste.rs had both downvoting and flagging until very recently.

I don’t think it had both, rather the flag button just looked like a “downvote” button. You still selected from the same reasons that flag presented. This change is the one that altered the styling I believe. There’s plenty of places that refer to it as “voting” in the code still, but unless I am mistaken there was only ever the one mechanism.

2. 7

A nice thing about downvotes is it lets people who simply don’t like something say “I don’t like this” without putting noise in the channel that’s used to catch abuse, spam, and so on.

So, the purpose of downvotes was emphatically not to express disagreement…it was to signal one of a handful of error conditions (bad information, factual inaccuracy, etc.). There is very little utility in seeing how many people merely disagree about something, especially given how fast people are to be mean to each other.

As a historical point, @jcs at one time disabled downvotes. It did not last more than a week or two before it was brought back.

1. 11

Sorry, I see how it was worded unclearly. I understand that the intention currently is to have no “disagree” button. I’m saying that having that button could reduce how often people flag comments they disagree with as having one of the error conditions, which they might do by by stretching definitions, picking a random reason, or just preferentially noticing real flaggable issues when they happen to dislike the content. (I think the unclarity comes from me using “downvotes” to mean “a no-reason downvote button” as opposed to flags as they exist–I edited to clarify above.)

It may be useful to let people push a button for disagreement even if your algorithms assign a low weight to those clicks. I say low weight rather than zero because having a 100%-placebo button in the UI raises other issues, even if moderators give zero weight to an easy expression of disagreement. I’d probably give a no-reason-given downvote nonzero weight in my head but I don’t expect others to adopt my weights.

Shorter, the idea is that people sometimes find a way to “downvote” for disagreement however you set up your UI, and you might get a net harm reduction by just giving them the button they want and then deciding what to do with the data.

1. 6

Ah, so kind of like the buttons on crosswalks?

Anyways, thank you for elaborating further. :)

1. 4

Very well said. Even if the system is not equipped with a “disagree” button, some users may desire one and use whatever means are at hand to express themselves. The phrase, “the purpose of a system is what it does,” comes to mind.

If existing mechanisms are not meant for use in disagreements, then we might ask if that is actually how they are being used.

2. 4

That’s nice, but you can’t make users behave the way you want them to behave, if you’re not going to actively act against it. People not only downvote because they disagree, but they’ll also flag things as spam or off-topic or “troll” just because they disagree, or feel you’ve insulted their particular religious (as in vimacs-vs-spaces) beliefs.

1. 6

I think I’d prefer a system that discourages misuse of flags. Maybe a similar message for people who flag a lot

1. 3

Sure, folks determined to game things would still flag. My theory is decently many who flag today are not determined to game things, and would click a down arrow for “just no” given the option. That’d reduce noise in the flag system even though it wouldn’t eliminate it. (And not saying you couldn’t do other changes too.)

2. 4

in fact, you can have placebo downvotes that show up for the person doing the downvoting but do not affect the vote count otherwise. that would keep (almost) everyone happy, i think.

3. 6

Agreed.

I think it’s fair to discuss whether the wording of the warning message can be improved, but various comments here that pretty much amount to “person X is popular, he should be exempt from the rules” are just tiring.

1. 8

How is being flagged a rule? It’s not a rule. That is the whole problem.

Anyone can flag any comment any time.

1. 2

Receive too many flags, get a notice. That’s the rule.

1. 9

That is not a rule in the sense of rule that can be followed by users. I did’t read a single comment suggesting that shouldn’t apply to some user because of its popularity, but rather many people pointing out that this is a clear case illustrating that such system is broken and should be fixed for all users.

2. 5

rather than continue to engage in a community they don’t fit into

Is lobster a community that accepts a plurality of opinion? If it is, then you will always have people disagreeing with you and therefore be liable to be reported by some of these people.

Democracy leads to lynching. Mass reporting is basically the lynching of internet persona. Any kind of automatic penalty that can be initiated via reporting is simply giving power to the angry mob. Autoban and autosuspend based on reports have always been abused.

1. 3

You’re still here, so evidently the suggestion doesn’t apply to you. You’re glad to contribute.

Why not show the warning to flag-happy people? I don’t have stats but I feel like that could be a better indicator of who doesn’t want to be here.

2. 21

Why do we need this passive aggressive banner anyway? It looks bad for Lobster, not for the person seeing it, and it’s not clear what problem it’s supposed to solve.

1. 19

We consistently see the vast majority of bad behavior come from the same few users. There’s a phenomenon, called the normalization of deviance, whereby people whose behavior is far away from the norm often genuinely don’t realize just how far it is. The banner, and the histogram that it links to, are intended as a way for people to self-correct. I personally am a big believer in giving people opportunities for redemption, and the histogram makes the point in a way that mere words can’t.

1. 6

far away from the norm

Far away from the norm doesn’t mean bad. You seem to be assuming that it is.

1. 8

I appreciate you raising that. On the contrary, I agree that sometimes people’s behavior is far from the norm for positive reasons. I do think that, even in those cases, everyone can benefit from a heads-up and a chance to reflect about whether it’s what they want.

1. 3

As in reflecting on do you want to be good when everybody else is bad?? Am I reading that right?

1. 7

Well, yeah. I know for a fact that people have left Lobsters for that reason (or at least, that’s how they see it). While I am sad when people leave, I don’t think anyone is obligated to stay somewhere that doesn’t make them happy. It would be wrong of me to try to trick or coerce people into staying when their free choice, with all the facts available, would be to leave.

I’m not really sure why you’re asking this, though. The flagging system doesn’t have the ability to determine who’s right and who’s wrong. Even if I wanted to take my personal concept of “good” and write code that ignores flags that aren’t “good”, there is no existing technology that can do anything like that. If that’s what you’re advocating for, feature request noted but I’m not able to do it.

1. 3

I’m not really sure why you’re asking this, though

I am not asking anything. I was commenting on an implicit bad argument you made (the implication that the norm is correct). You decided to double down by arguing that even if somebody was correct, they should ‘reflect about whether it’s what they want’. I simply then pointed out that this doesnt reinforce your original argument, because it itself is bad, because having a bad feature (aggressive warning about being reported) is not validated by having that feature also be useful to do useless thing (letting people reflect on whether they should be good, because the answer is always yes).

I don’t think anyone is obligated to stay somewhere that doesn’t make them happy.

Of course not. The point is that assuming that lobsters want a community that engage in meaningful discussion, then it should make people want to stay and be happy, that is a worthy goal. all of this is following your argument for the banner, the argument which I consider to be invalid, as pointed out.

1. 5

Being distant from the norms can hold little moral judgment and yet be relevant when operating in a constructed community.

1. 2

Okay, I guess that’s fair.

I think we disagree about something in there, but it sounds like figuring out what isn’t a high priority for either of us? I appreciate the discussion, anyway.

2. 1

I at least didn’t read that implication into the post. Not all ways of being far from the norm are likely to result in seeing the banner. It isn’t a perfect system but it’s definitely not symmetric, so that assumption isn’t necessarily required.

3. 2

Thanks for clarifying. But I still think it doesn’t make Lobster looks good because this feature implies that “deviant” behaviour should be corrected and can be automatically detected, and I expect most people (especially in tech) aren’t comfortable with either of these.

I’m wondering what’s the worse that could happen if the banner is not displayed? Maybe the person who’s being flagged all the time will get the point and adjust their behaviour, or maybe they don’t need to adjust anything and it’s other people that will start seeing that perhaps that person has a point. I also believe in giving people a second chance but I don’t think the solution to everything has to be algorithms, sometimes simple human interactions are preferable.

1. 4

The hypothesis that motivates the banner and histogram is that, by displaying them, those conversations about what to adjust will happen sooner and be more productive. The goal is to head off situations where somebody doesn’t believe mods when we tell them something needs to change, and they ultimately leave the site. It may be right or wrong but that’s the motivation.

2. 1

Do you think that the banner has an observable positive effect? Do you see a change in behavior from the few users you say produce the majority of bad behavior? As friendlysock says above, they see the banner nearly all the time. Do you think they have been falsely-flagged by this system? If so, can you estimate how high the false-positive rate of showing this banner? If friendlysock has been appropriately warned by this system, have you seen any change in their behavior onsite?

If it is truly only a few users producing undesirable behavior, could it be more effective to simply reach out to those users personally?

1. 4

I’m not going to publicly discuss my personal feelings about specific community members. It wouldn’t be fair to anyone.

I don’t have a methodology to conclusively determine how much influence the banner has had, but the site has a lot less flagging in general than it did a couple years ago. I take that to be an indicator that things have calmed down significantly, and I do think that’s good.

I do reach out to people personally from time to time, as do the other mods. The banner and histogram were created to address a shortcoming in that approach, where sometimes people aren’t willing to listen to criticism unless they’re presented with data that validates it. I’m skeptical of going fully data-driven for everything, but I think it’s nice to have at least this one particular metric for those who find it easier to take concerns seriously when there’s a metric.

4. 20

Additional suggestion to all the previous improvements: make that banner a thing that’s moderator-initiated. I of course don’t have statistics on how many people see it (I never got it, so I just learned about it now, anecdotally) but I could imagine that this is rare enough that moderators could just evaluate sending this message to users by hand after the system flags the user internally.

1. 6

I of course don’t have statistics on how many people see it

I don’t have that statistic either, but as I posted elsewhere, only 13 people have been flagged more than me in the last month and I haven’t seen it. Only 7 people have been flagged 10 or more times. I’d therefore expect that trimming it down to 5 people that moderators need to take a look at per month is quite feasible.

If even that is too much of a load for the moderators, then it might be possible to select a dozen people who have been active recently but not participated in any of the flagged threads to do a triaging pass. At that point, having someone reach out to the person and say ‘you’ve had a lot of things flagged recently, is there anything going on in your life that you want to talk about?’ would be a fairly low load.

1. 3

You can a chart that gives you a clear idea how many people see it at `/u/you/standing`. So you can see it at /u/david_chisnall/standing and I can see it at /u/hoistbypetard/standing. The red Xs should represent users seeing the warning.

I’d never suggest that someone else should just add a thing to their plate, but it does seem like the kind of thing where, if there were a few active moderators, requiring someone to approve the red box might be an improvement for a relatively low effort. That feels like the kind of statement I should only make if I’m accompanying it with a patch, though, and I can’t offer that right now.

2. 15

Yep, I think the current wording will probably trigger exactly the wrong people “have I overstepped a border? Probably” and the people who are actively engaging in a little bit of flaming won’t be bothered.

1. 7

Following on the “let’s get shit done” vibe, is there an open issue for discussing this in more detail on github? I did a quick lookup and didn’t find any, would be cool if it was posted here (even better if as a top comment)

1. 5

Good idea, thanks! (Obvious in retrospect, like many good ideas.) I created an issue.

2. 4

I mildly agree with both of those things, but I also think that the suggestion further down the thread to re-introduce downvoting as “non-bannable feedback” could address the issue more directly.

Some people may misuse flags and downvotes, but overall I think these guidelines are intuitive and easy to remember:

• upvote: this should go higher on the page
• downvote: this should go lower on the page
• flag: if the user continues this behavior, they should eventually be banned
1. 2

In my post I’m trying to focus on an easy thing to do to help reduce this problem. I have the impression that rewording the banner is something that’s reasonably easy, while “let’s reintroduce downvotes” or “the banner should be approved by moderators before being shown” are important change in process that require a lot of discussion. They may be excellent ideas, but it’s a lot more work to discuss and turn them into action, so I think it’s best not to mix the two kind of suggestions if we want to change anything at all in the short term; they should be discussed separately, and maybe differently. (I could just send a PR to rephrase the banner; any of the other suggestions requires more discussion with contributors, etc.).

1. 1

Yes, it makes sense to do the easy things first. My impression was that we had downvotes several months ago, so it shouldn’t that big a change to restore them, but I’m not involved so I could be wrong.

I was mystified by the removal of downvotes, and never saw the rationale for it. I saw one thread discussing it 3 or 6 months after they were removed, which still didn’t answer my questions about why they were removed. It made the site worse for me, but not enough to really complain about it. I know this is an all-volunteer effort and I didn’t have time to look at it myself.

1. 1

Nitpicky, but please don’t use mean/sigma to find outliers in your metrics. Use a robust indicator like median/MAD or, if your distribution is asymmetric (and it probably is) look at a histogram.

1. 4

I resonate with so much of this thread:

1. The attraction to APL/J that never really sticks
2. The sad if inevitable uncovering of the limitations to the things that are always hyped up about APL/J
3. The vague, hard to express sensation that there’s something there [in that language, paradigm, algorithm…] that feels important!

In the last two years my sense of (3) has grown more and more. I don’t know what to do with it quite yet.

1. 1

I get the sense that there might be some languages or paradigms that might work well (best?) in an embedded context. Perhaps embedding APL into another language as a DSL or a DLL might provide a way to use the language for what it’s good for and avoid the difficult parts.

1. 3

I haven’t tried using it this way, but GNU APL can be built as a library and linked into another project as an embedded scripting language DSL.

1. 2

Posts 19-21, about the folly of Unicode for shortening programs, echo strongly for me. In e.g. Haskell and Agda, there are options for allowing Unicode to intermix with an otherwise-ASCII-only syntax, creating a rich experience. A Lojbanist took this too far recently, creating unreadable emoji soup.

1. 1

You don’t have to take it too far, though. `∀(x ∈ S: x² ≤ 10)`?

1. 3

It doesn’t generalize very well. There are Unicode superscripts for each of the digits but to find a superscript “x”, e.g. 2ˣ, you have to find it in a completely unrelated block of characters, which also still doesn’t have the complete superscript alphabet.

1. 1

Because that was the wrong generalization. Function application should still be explicit instead of using a whole new unicode block for a single power function. Superscripts are better used for denoting special entities like C¹⁴.

1. 2

I wish we could do away with this archaic version of Game of Life for APL demos. Modern APLs have convolution operators that are more suited to that type of problem than wastefully creating multiple rotated copies of the input array.

1. 2

Also, the conventional version has a lot of problems.

1. 4

APL isn’t elegant for all problems, but I don’t think all of your examples demonstrate that point well.

For example, I was able to quickly adapt the standard solution in J to solve Brian’s Brain. Per the article you posted, the “APL way” is (mostly) not to introduce new abstractions for re-use, but to deploy variations of idioms that fluent programmers are expected to be familiar with. In this case, I straight copied `[:+/(>,{;~i:1)|.!.0` for the neighbor counts and then used the standard trick of multiplying by 0-1 to do filtering.

Similarly, you can write a J/APL solution that doesn’t waste as much memory by using a list of complex numbers to represent live cells, then adding `1 1j1 0j1 _1j1 _1 _1j_1 0j_1` to each of them (using table `/`) and filtering the results by counts and comparison with the original list for new live cells. I suspect a similar approach can be adapted for hexagonal tiling. I think I’ve done this before at some point.

And in other cases the natural solution isn’t obvious, but does exist.

The places I’ve seen APL struggle with no way out are procedural algorithms that rely heavily on state mutation. Of course, both J and APL do offer standard loops and variables, so you can always drop down when array thinking won’t solve your problem. I don’t think the claim is that it’s a panacea, only that it is very powerful across a wide variety of real problems, and even more of those the more fluent you become.

1. 5

I appreciate this run through. My continually relevant tweet from 6 years ago is relevant once again, https://twitter.com/losvedir/status/636034419359289344.

I will say that one area that the array language influence “stuck” was with CSS. For a while I preferred one line class definitions, with no line breaks between related classes, eg:

``````.foo{display: flex; border: 1px solid #ddd;}
.foo-child{flex: 0 0 100; padding: 1rem;}
``````

But then that made me more receptive to tailwind style utility CSS, so that’s where I am now.

But array languages are so cool, and I really wonder how much is syntactic (terseness as a virtue, all these wonderful little operators), and how much is semantic (working on arrays, lifting operators to work at many dimensions). What would a coffeescript like transpiler from more traditional syntax to, say kdb/q, be like?

1. 6

IME, the real magic of APL, and what the numerous APL-influenced array languages have consistently lost in translation, are the concatenative, compositional, functional operators that give rise to idiomatic APL. They have taken the common usecases, but forgone the general ones. For example, numpy provides cumsum as a common function, but APL & J provide a more general prefix scan operator which can be used with any function, no matter whether primitive or user-defined, giving rise to idioms like “running maximum” and “odd parity” to name just a couple. Likewise, numpy has inner but it only computes the ordinary “sum product” algorithm while APL & J have the matrix product operator that affords the programmer the ability to easily define all sorts of unusual matrix algorithms that follow the same inner pattern.

This is not even to mention the fantastic sorts of other operators, like the recursive power of verb or the sort-of-monadic under that AFAICT have no near equivalent in numpy.

1. 1

Is there a simple way for other languages to replicate the success, or do the designers just need to be brilliant?

1. 7

I doubt brilliance has much to do with it. It’s likely more about exposure to the concepts coinciding with the motivation required to model them in a language or library. Especially in a way that’s accessible to people who don’t have previous exposure. Learning the concepts thoroughly enough to make it simple, and doing the work required to create an artifact people can use and understand is really difficult.

You see similar compositional surprises when looking at some of the Category Theory and Abstract Algebra inspired Haskell concepts. I imagine the current wave of “mainstream” interest in Category Theory will result in these ideas seeping into more common usage, and exposed in ways that don’t require all the mathematical rigor.

It’s important to realize that APL-isms are beautiful, but they are especially striking to people because it’s new to them. Set theory, the lambda calculus, and relational algebra are just some things that have similarly inspired in the past (and continue to do so!) that have spread into common programming to the extent that casual users don’t realize they came from formalized branches of mathematics. In my opinion this is a good thing!

Another exciting thing happening right now is the re-discovery of Forth. It has similar compositional flexibility, but goes about things in a very different way that corresponds to Combinatory logic. I would expect some people are going to reject the Abstract Algebra/Category Theory things as “too far removed from the hardware”, but be jealous of the compositional elegance. This will result in some very excited experimentation with combinatory logic working directly on a stack. Not that this hasn’t been happening in the compiler world with stack machines for decades…but it’s when non-specialists get ahold of things that innovation happens and things get interesting.

1. 21

Please don’t bring this toxicity over to lobste.rs. Let it live and die on the orange site.

1. 19

I’ve had a few interesting discussions about our tribal and memetic expectations as exemplified by this repository. Sometimes, when somebody does something unbearably cringy in the name of what they think is supremely excellent taste, it is an opportunity to delve deeper into their implicit beliefs.

Nowhere in this repository is it ever imagined that C is not an appropriate language for today’s coders. Rather, the machismo which is required to write fresh C code against an ever-steeper gradient of better toolchains for higher-level languages is also the same machismo which is stoked by the underlying family of memes used to theme and decorate the repository.

Sometimes a project’s only purpose is to serve as a warning to others. Even in those moments, let us learn something from the precise nature of the warning.

1. 22

Can you define what you mean by ‘toxicity’? I don’t see anything ‘toxic’ in this repository.

1. 5

I don’t like the way its naming and meming relies on using virgin man vs not as a shorthand for bad vs not, however ironically.

It’s a cool trick. Why couple it with a bad meme?

1. 19

The meme is incredibly funny, that’s why.

2. 21

Toxicity? Orange site? I didn’t get any of that.

I could agree that the theming is a little overboard, not because of bad taste, but because it detracts from the technical side, as speps said.

Code needs to market itself, and it seems “skullchap” had some fun in that department.

1. 11

It seems interesting from a technical perspective but I agree on the bad taste aspect, it detracts from the technical side.

1. 8

I think we need a new flag, because this isn’t really spam or off-topic: I suggest “bad taste” ;)

1. 27

The best way to deal with content that’s on-topic but otherwise not very valuable is simply to ignore it.

A comment, btw, counts as an upvote, at least as far as visibility on the front page is concerned.

1. 16

Or maybe some more tags. “based” and “redpilled” come to mind :-D

Alright, I’ll see myself out.

1. 5

Agreed. I marked it off-topic. Whatever your taste or politics, this repository was built simply “for the lulz”. It’s not tech, it’s a social experiment with code.

While there are many social topics in the tech world, that’s generally not what lobste.rs is good for.

1. 53

It’s tech, you and others just seem to be distressed by the cultural shibboleths it touches.

We have articles here that are dripping with furry stuff that are routinely defended from fursecution…a mainstream meme plus cool hack isn’t outside our Overton window for me.

1. 10

Labeling something “toxic” with the intent of banning it is as much of a cultural shibboleth as any of the memes in the article.

1. 4

The furry stuff is a positive expression of someone’s own personal identity, while the chad/virgin meme is a ridiculous objectification of others for being ingroup/outgroup. Do you see a difference?

1. 4

Yeah, what’s up with all the furry stuff?

2. 13

this repository was built simply “for the lulz”. It’s not tech,

It’s not? I’m confused. I haven’t tried compiling it, but there’s a C file in there with a bunch of complex-looking code in it. Does that count as tech?

I don’t know if anyone would use this library for reals (I sure hope not), but the same goes for stuff like Brainfuck or (IMHO) a 6502 assembler. There’s a lot of stuff posted here that is, at some level, done for teh lulz.

1. 8

this repository was built simply “for the lulz”. It’s not tech, it’s a social experiment with code.

A lot of software is written “just because” or “for the lulz”. A lot of good software is even written for this express purpose— Linux started out as a hobby project, no?

I also don’t see what makes this a social experiment. What’s the experiment? The readme being silly? Is that an experiment?

1. 4

Linus Torvalds biography is called “Just for fun”, because that is why he wrote Linux. That is how older people say “for the lulz”. Is Linux now also off-topic?

There is some actual C macro (ab-)use in there, that is interesting to look at. It is def. tech.

Why are you so triggered by this stupid meme?

2. 6

The shitposting C chad versus the virgin crustacean.