1. 4

    Great to see that data get used. One question I had after skimming the paper: how much of this performance improvement do you attribute to Noria’s data-flow computation? One of the frustrations comparing these things is that MySQL has a lot of features that Noria presumably doesn’t, makes it sort of apples-and-oranges. But I really like this model of incremental computation, I see it as a larger popular trend in programming tools.

    1. 3

      Hmm, I’m not entirely sure I understand the question, but I think maybe there’s a misunderstanding about how the system works. Most of the performance improvement in Noria is due to the fact that basically all SELECTs are now direct cache hits. The data-flow is just the way that we ensure that the results for those SELECT queries remain updated as the underlying data changes (e.g., as new votes are added). This design basically decouples the performance of reads and writes: reads are always fast, and the write throughput is determined by how many things the write touches; we’re nowhere near the limit of how many reads per second we could do for the lobsters queries (see the vote results). As queries get more complicated (and again, we can support all the queries in the lobsters source code), reads do not get any slower, only writes do. It is true that now the writes are slower, but they are also rarer. In the Lobsters workload at the moment, the biggest bottleneck is the write path for updates to read_ribbons, and that’s what’s preventing us from scaling beyond 5x MySQL. Sharding that write path may be the way to resolve that issue down the line.

      As for feature parity, I don’t know exactly what you’re referencing? Is there a particular feature you’re worried about Noria not having that you rely on for MySQL? Not sure if it came across in the paper, but you can take unmodified applications that just use mysql client libraries and just plug’n’play them with Noria. At least that’s the idea modulo our SQL parser and query planner still not being quite as mature as MySQL’s.

      1. 4

        So Noria maintains materialized views, sort of like flexviews but with automatic refreshing or like pipelinedb but base data is permanent (table) rather than ephemeral (stream). Also reminds me somewhat of ksql. And since it is the database, the application doesn’t need to handle complicated and error-prone cache invalidation (e.g. in the typical MySQL + memcache scenario). Pretty neat!

        I had the same question about apples-to-oranges comparison though. For example, transaction support, foreign keys, different index types, triggers, rocksdb vs innodb implications.

        1. 3

          Yup, you are totally right that there are features of more traditional databases that we do not yet support. This is still a research prototype, so it’s focused on the research problems first and foremost. We don’t believe any of those additional features to be fundamentally impossible in the Noria paradigm though — for example, we’re designing a scheme for adding transactions, and we believe we can do it without adding much overhead to query execution in the common case!

          Some of these other features are also really optimization details. For instance, since Noria knows the application’s queries, it could automatically choose indexes that fit the query load (even though currently it only uses hash indexes). Similarly, RocksDB vs InnoDB shouldn’t matter to the application. We use RocksDB only for storing the base table data, not for storing anything else, so it’s mostly just there for persistence, and rarely affects performance.

          As for foreign keys and triggers, those should be pretty easy to add, and mostly just need engineering, not research. In a sense, triggers are really just additional operators in the graph, so they’re almost a non-feature in Noria.

          1. 3

            You may also find the discussion on Reddit interesting.

          2. 2

            My question isn’t about how the system works, it’s about the breadth of MySQL, which pays a performance cost for lots of features I presume Noria doesn’t have. Multi-master setups, sharding, charset collations, many more data types, support for at least five operating systems, date and time functions, multiple storage formats, a million things. Even if Lobsters doesn’t use them, some of those are going to result in conditionals on the hot path to serving even very simple, performant queries like select * from users where id = 123 and account for some of the performance difference. I say it’s sort of an apples-and-oranges comparison because Noria and MySQL have such different featuresets - if it were possible to compile a version of MySQL that dropped support for every feature Lobsters doesn’t use, I wonder if that wouldn’t be in the neighborhood of 5x faster. I have so little intuition for it I wouldn’t be surprised at 1.01x or 20x.

            Edit: ah, and after I hit post I reloaded the page to see @tobym made this point and you already responded to it. I’ll check out the reddit link. :)

            1. 3

              In addition to my response to @tobym, let me try to address some of your specific concerns too. First, Noria already supports multi-machine distribution and sharding, and replication is already nearly done. Noria is also more flexible than MySQL in its data types, since it doesn’t have strict column typing. If we did apply the same schema strictness as MySQL, that would improve our performance, since we could specialize data-structures to known types. While it is true that we don’t support as many data types as MySQL, adding news ones is pretty straightforward, and we already support quite a few. Similarly, adding date and time functions should be straightforward – they are just new projection and filter operations. Noria should also run without modifications on Linux, macOS, and Windows.

              As for multiple storage formats, Noria is, in a sense, arguing that you as the developer shouldn’t have to think about that. You should tell the database what your queries are, and it should determine how best to persist and cache the data and the query results. Are there particular features associated with the storage systems that you had in mind?

              You are right though that MySQL does more than Noria does, and that that adds overheads that Noria does not have in some cases. However, most of Noria’s performance advantage comes from the model — computing on write instead of read — as opposed to implementation. MySQL fundamentally has to compute things on reads, whereas Noria does not, and with most operations being reads, that translates to speed-ups that MySQL cannot recover. It would be great to disable lots of MySQL features, but it is unlikely to change the picture much due to this fundamental design difference.

              The one exception to this is transactions: it could be that transactions are just so expensive to provide, that the MySQL was is just way faster than anything you could achieve in the Noria paradigm. We don’t believe this to be the case though, as we already have a design sketch that adds transactions and strongly consistent reads to Noria while introducing nearly no overhead in the common case.

        1. 2

          Dude, that’s awesome! Great work! I see you published the source. Were there any patents on this or that not a concern?

          1. 3

            Thanks! We certainly haven’t filed for any patents, and I’m not aware of any patents relevant to this. As you can see from the paper’s related work section, this does build on a lot of insights from other fields and systems, but I think it nonetheless carves out its own little nook of data-flow and database research that others haven’t really explored before. As for publishing it as open-source, that has been my goal all along. I don’t personally have any desire for commercializing this, though I also think it’s something that would work very well under a Redis-like or Postgres-like model where it could be a serious open-source production system with enterprise support.

          1. 3

            To everyone who’s curious about the results of this, and to @pushcx in particular, the paper is now published! https://jon.tsp.io/papers/osdi18-noria.pdf

            1. 1

              Thanks for publishing it. This is a really, old thread, though. I suggest you submit it as its own story with authored by and a reference to this thread in text field. Then it will get more attention.

              1. 2

                Yeah, @pushcx made the same suggestions privately. That was the plan all along, but I wanted to wait until we’d held the conference talk (which we just did!) and polished up the codebase + docs a little. Submitted it now as https://lobste.rs/s/uxgjha/noria_dynamic_partially_stateful_data !

            1. 7

              There seems to be more “intermediate” level Rust posts recently. OP is a good example.

              1. 8

                It’s true, and great to see! It might be in response to the Rust 2018 roadmap, which specifically highlights:

                One of the strongest messages we’ve heard from production users, and the 2017 survey, is that people need more resources to take them from understanding Rust’s concepts to knowing how to use them effectively. The roadmap does not stipulate exactly what these resources should look like — probably there should be several kinds — but commits us as a community to putting significant work into this space, and ending the year with some solid new material.

                <plug shameless=true> For example, I’ve been doing Rust live-coding sessions where we build non-trivial (and useful) Rust crates (e.g., I recently did one on writing an asynchronous SSH client), and those have been pretty well received. It seems like we’re getting more and more Rustaceans who want to try Rust for “real things”, and that’s pretty exciting! It’s good to see the supply meeting that demand.

                1. 6

                  (OP) I think it’s a virtuous cycle. I got into Rust because I read some good entry level posts that talked about how it was ready for prime time.

                1. 5

                  Results here. Thanks for your patience, @jonhoo, I was traveling.

                  SELECT ROUND(upvotes + downvotes, -2) AS bucket, COUNT(id), RPAD('', LN(COUNT(id)), '*') FROM stories GROUP BY bucket
                  
                  +--------+-----------+------------------------------+
                  | bucket | COUNT(id) | RPAD('', LN(COUNT(id)), '*') |
                  +--------+-----------+------------------------------+
                  |      0 |     40286 | ***********                  |
                  |    100 |       360 | ******                       |
                  |    200 |         2 | *                            |
                  |    300 |         2 | *                            |
                  +--------+-----------+------------------------------+
                  4 rows in set (0.30 sec)
                  

                  SELECT ROUND(comments_count, -2) AS bucket, COUNT(id), RPAD(’’, LN(COUNT(id)), ‘*’) FROM stories GROUP BY bucket;

                  +--------+-----------+------------------------------+
                  | bucket | COUNT(id) | RPAD('', LN(COUNT(id)), '*') |
                  +--------+-----------+------------------------------+
                  |      0 |     40482 | ***********                  |
                  |    100 |       165 | *****                        |
                  |    200 |         3 | *                            |
                  +--------+-----------+------------------------------+
                  3 rows in set (0.29 sec)
                  

                  SELECT ROUND(counted.cnt, -2) AS bucket, COUNT(user_id), RPAD(’’, LN(COUNT(user_id)), ‘*’) FROM ( SELECT user_id, COUNT(id) AS cnt FROM votes GROUP BY user_id) AS counted GROUP BY bucket;

                  +--------+----------------+-----------------------------------+
                  | bucket | COUNT(user_id) | RPAD('', LN(COUNT(user_id)), '*') |
                  +--------+----------------+-----------------------------------+
                  |      0 |           4439 | ********                          |
                  |    100 |            562 | ******                            |
                  |    200 |            206 | *****                             |
                  |    300 |            131 | *****                             |
                  |    400 |             92 | *****                             |
                  |    500 |             45 | ****                              |
                  |    600 |             28 | ***                               |
                  |    700 |             31 | ***                               |
                  |    800 |             29 | ***                               |
                  |    900 |             32 | ***                               |
                  |   1000 |             14 | ***                               |
                  |   1100 |             23 | ***                               |
                  |   1200 |             18 | ***                               |
                  |   1300 |             18 | ***                               |
                  |   1400 |             13 | ***                               |
                  |   1500 |             10 | **                                |
                  |   1600 |              3 | *                                 |
                  |   1700 |              7 | **                                |
                  |   1800 |              8 | **                                |
                  |   1900 |              6 | **                                |
                  |   2000 |              3 | *                                 |
                  |   2100 |              3 | *                                 |
                  |   2200 |              4 | *                                 |
                  |   2300 |              6 | **                                |
                  |   2400 |              4 | *                                 |
                  |   2500 |              2 | *                                 |
                  |   2600 |              3 | *                                 |
                  |   2700 |              2 | *                                 |
                  |   2800 |              2 | *                                 |
                  |   2900 |              3 | *                                 |
                  |   3000 |              2 | *                                 |
                  |   3100 |              3 | *                                 |
                  |   3200 |              2 | *                                 |
                  |   3300 |              2 | *                                 |
                  |   3400 |              1 |                                   |
                  |   3600 |              1 |                                   |
                  |   3800 |              1 |                                   |
                  |   3900 |              1 |                                   |
                  |   4000 |              1 |                                   |
                  |   4100 |              3 | *                                 |
                  |   4200 |              1 |                                   |
                  |   4300 |              6 | **                                |
                  |   4400 |              1 |                                   |
                  |   4500 |              2 | *                                 |
                  |   4600 |              1 |                                   |
                  |   4800 |              1 |                                   |
                  |   5000 |              1 |                                   |
                  |   5100 |              1 |                                   |
                  |   5400 |              1 |                                   |
                  |   5500 |              1 |                                   |
                  |   5800 |              2 | *                                 |
                  |   5900 |              1 |                                   |
                  |   6000 |              1 |                                   |
                  |   6400 |              2 | *                                 |
                  |   6800 |              1 |                                   |
                  |   7100 |              1 |                                   |
                  |   7700 |              1 |                                   |
                  |   8400 |              1 |                                   |
                  |   8700 |              1 |                                   |
                  |   8900 |              1 |                                   |
                  |  10100 |              1 |                                   |
                  |  10900 |              1 |                                   |
                  |  13500 |              1 |                                   |
                  |  14500 |              1 |                                   |
                  |  15100 |              1 |                                   |
                  +--------+----------------+-----------------------------------+
                  65 rows in set (0.15 sec)
                  

                  nginx grep:

                  This log started 2018-02-11 04:40:31 UTC. I edited some tokens and login information out, replacing with [elided].

                  1133494 GET /stories/X/Y
                   590907 GET /
                    71829 GET /comments
                    19150 GET /comments.rss
                    12020 POST /comments/X/upvote
                     9033 POST /stories/X/upvote
                     6126 POST /comments
                     4591 GET /login
                     3681 GET /comments/X/reply
                     1660 POST /login
                     1429 GET /comments/X/edit
                     1301 POST /comments/X/2.0"
                     1174 POST /stories/X/save
                     1019 POST /stories
                      929 POST /stories/X/hide
                      637 POST /comments/X/downvote
                      535 GET /comments/X/2
                      390 POST /stories/X/2.0"
                      224 POST /comments/X/unvote
                      202 GET /comments/X/3
                      198 GET /comments/X/2.0"
                      190 POST /stories/X/downvote
                      174 GET /login/forgot_password
                      150 POST /stories/X/suggest
                      148 POST /stories/X/unvote
                      148 GET /login/2fa
                      132 POST /login/2fa_verify
                      109 POST /login/reset_password
                       96 GET /comments/X/4
                       94 POST /stories/X/unsave
                       63 POST /comments/X/delete
                       55 POST /logout
                       54 GET /comments/X/5
                       37 POST /stories/X/unhide
                       34 GET /comments.rss?token=[elided]
                       29 GET /comments/X/6
                       27 POST /login/set_new_password
                       19 POST /comments/X/1.1"
                       19 GET /comments/X/7
                       19 GET /comments/X/1.1"
                       12 POST /
                       12 GET /comments/X/3170
                       10 POST /comments/X/undelete
                        9 GET /comments/X/8
                        8 GET /comments/X/9
                        8 GET /comments/X/10
                        7 GET /logout
                        5 GET /login/reset_password
                        5 GET /comments/X/11
                        4 GET /comments/X/
                        2 POST /stories/X/1.1"
                        2 POST /comments?RzVI%3D3496%20AND%201%3D1%20UNION%20ALL%20SELECT%201%2CNULL%2C%27%3Cscript%3Ealert%28%22XSS%22%29%3C%2Fscript%3E%27%2Ctable_name%20FROM%20information_schema.tables%20WHERE%202%3E1--%2F%2A%2A%2F%3B%20EXEC%20xp_cmdshell%28%27cat%20..%2F..%2F..%2Fetc%2Fpasswd%27%29%23
                        2 GET /comments/X/5000
                        2 GET /comments/X/12
                        1 POST /stories/X/do_we_need_move_away_from_elm
                        1 POST /login?utf8=%E2%9C%93&authenticity_token=[elided]&email=[elided]&password=[elided]&commit=Login&referer=https%3A%2F%2Flobste.rs%2Fs%2Fcgqz3p%2Fdo_we_need_move_away_from_elm
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /login/set_new_password?token=[elided]
                        1 GET /comments/X/7000
                        1 GET /comments/X/6000
                        1 GET /comments/X/5700
                        1 GET /comments/X/5600
                        1 GET /comments/X/5570
                        1 GET /comments/X/5559
                        1 GET /comments/X/5558
                        1 GET /comments/X/5555
                        1 GET /comments/X/5550
                        1 GET /comments/X/5500
                        1 GET /comments/X/20000
                        1 GET /comments/X/200
                        1 GET /comments/X/20
                        1 GET /comments/X/14
                        1 GET /comments/X/13
                        1 GET /comments/X/10000
                        1 GET /comments.rss?token=[elided]
                  
                  1. 1

                    This is fantastic, thank you!

                    Looks like my log grepping wasn’t perfect, but we can clean up the remainder pretty easily. Feel free to edit out any lines from the nginx log with a count < 10. I posted a slightly updated command in a comment a few days ago that includes /recent and /u and makes some other fixes – if you could run that one (mostly for the /recent numbers), that’d be awesome.

                    It looks like the comment and count numbers are a little smaller than I expected, which makes the distribution hard to infer from the few resulting data points. Any chance you could re-run the first two commands with ROUND(..., -1) instead (rounds to the nearest 10 instead of nearest 100)?

                    1. 2

                      The updated nginx grep didn’t work. My sed (identifies itself as sed (GNU sed) 4.4) errored on three latter instances of //. I took a stab at correcting it but couldn’t immediately spot what it should’ve been.

                      And, sure, here’s the first two with the smaller buckets:

                      +--------+-----------+------------------------------+
                      | bucket | COUNT(id) | RPAD('', LN(COUNT(id)), '*') |
                      +--------+-----------+------------------------------+
                      |      0 |     16724 | **********                   |
                      |     10 |     16393 | **********                   |
                      |     20 |      4601 | ********                     |
                      |     30 |      1707 | *******                      |
                      |     40 |       680 | *******                      |
                      |     50 |       281 | ******                       |
                      |     60 |       128 | *****                        |
                      |     70 |        60 | ****                         |
                      |     80 |        35 | ****                         |
                      |     90 |        16 | ***                          |
                      |    100 |         4 | *                            |
                      |    110 |         4 | *                            |
                      |    120 |        10 | **                           |
                      |    130 |         1 |                              |
                      |    140 |         2 | *                            |
                      |    160 |         1 |                              |
                      |    210 |         1 |                              |
                      |    250 |         1 |                              |
                      |    290 |         1 |                              |
                      +--------+-----------+------------------------------+
                      19 rows in set (0.32 sec)
                      
                      
                      +--------+-----------+------------------------------+
                      | bucket | COUNT(id) | RPAD('', LN(COUNT(id)), '*') |
                      +--------+-----------+------------------------------+
                      |      0 |     33974 | **********                   |
                      |     10 |      4831 | ********                     |
                      |     20 |      1029 | *******                      |
                      |     30 |       401 | ******                       |
                      |     40 |       193 | *****                        |
                      |     50 |       103 | *****                        |
                      |     60 |        50 | ****                         |
                      |     70 |        24 | ***                          |
                      |     80 |        18 | ***                          |
                      |     90 |        10 | **                           |
                      |    100 |         5 | **                           |
                      |    110 |         1 |                              |
                      |    120 |         5 | **                           |
                      |    130 |         2 | *                            |
                      |    140 |         1 |                              |
                      |    150 |         2 | *                            |
                      |    170 |         1 |                              |
                      +--------+-----------+------------------------------+
                      17 rows in set (0.31 sec)
                      
                      1. 1

                        Huh, seems like something weird happened when I copied it. Here, try this one?

                        grep -vE "assets|fetch_url_attributes|check_url_dupe|set_new_password|reset_password|forgot_password" lobsters.log
                         | sed -e 's/.*\(GET\|POST\)/\1/'
                               -e 's/\/\(s\|stories\)\/[^\/ ]*/\/stories\/X/'
                               -e 's/\/u\/[^\/ ]*/\/u\/X/'
                               -e '/^GET / s/X\/.\+/X/'
                               -e 's/\/comments\/[^\/ ]*/\/comments\/X/'
                         | awk '/^(GET|POST)/ {print $1" "$2}'
                         | grep -E ' /(stories|comments|login|logout|u/|$)'
                         | sort | uniq -c | sort -rnk1,1
                        
                        1. 2
                          1139580 GET /stories/X
                           594323 GET /
                           159959 GET /u/X
                            72320 GET /comments
                            19287 GET /comments.rss
                            12088 POST /comments/X/upvote
                             9081 POST /stories/X/upvote
                             6160 POST /comments
                             4597 GET /login
                             3704 GET /comments/X/reply
                             1663 POST /login
                             1433 GET /comments/X/edit
                             1322 POST /comments/X
                             1188 POST /stories/X/save
                             1022 POST /stories
                              932 POST /stories/X/hide
                              641 POST /comments/X/downvote
                              542 GET /comments/X/2
                              395 POST /stories/X
                              224 POST /comments/X/unvote
                              219 GET /comments/X
                              207 GET /comments/X/3
                              192 POST /stories/X/downvote
                              153 GET /login/2fa
                              150 POST /stories/X/suggest
                              149 POST /stories/X/unvote
                              137 POST /login/2fa_verify
                              101 GET /comments/X/4
                               94 POST /stories/X/unsave
                               63 POST /comments/X/delete
                               55 POST /logout
                               55 GET /comments/X/5
                               37 POST /stories/X/unhide
                               34 GET /comments.rss?token=[elided]
                               29 GET /comments/X/6
                               19 GET /comments/X/7
                               12 POST /
                               12 GET /comments/X/3170
                               10 POST /comments/X/undelete
                          
                          1. 1

                            Oops, looks like I forgot /recent again in that query :( Could you run:

                            grep -vE "assets|fetch_url_attributes|check_url_dupe|set_new_password|reset_password|forgot_password|token" lobsters.log
                             | sed -e 's/.*\(GET\|POST\)/\1/'
                                   -e 's/\/\(s\|stories\)\/[^\/ ]*/\/stories\/X/'
                                   -e 's/\/u\/[^\/ ]*/\/u\/X/'
                                   -e '/^GET / s/X\/.\+/X/'
                                   -e 's/\/comments\/[^\/ ]*/\/comments\/X/'
                             | awk '/^(GET|POST)/ {print $1" "$2}'
                             | grep -E ' /(stories|comments|login|logout|u/|recent$)'
                             | sort | uniq -c | sort -rnk1,1
                            

                            And also let me know the start time of the log and the time you run the query (so I have an idea of the interval)?

                            The workload generator is also shaping up over here :)

                            1. 2

                              Timestamp at the start is 11/Feb/2018:04:40:31 +0000, end is 27/Mar/2018:01:26:49 +0000. I’m glad to see these stats going into your work, and I look forward to seeing your finished work submitted as a story.

                              1615623 GET /stories/X
                               193915 GET /u/X
                               105680 GET /comments
                                27731 GET /comments.rss
                                23155 GET /recent
                                18235 POST /comments/X/upvote
                                13730 POST /stories/X/upvote
                                 9141 POST /comments
                                 7137 GET /login
                                 5572 GET /comments/X/reply
                                 2512 POST /login
                                 2233 GET /comments/X/edit
                                 2055 POST /comments/X
                                 1735 POST /stories/X/save
                                 1522 POST /stories
                                 1422 POST /stories/X/hide
                                 1159 POST /comments/X/downvote
                                  801 GET /comments/X/2
                                  600 POST /stories/X
                                  391 POST /comments/X/unvote
                                  347 POST /stories/X/downvote
                                  340 GET /comments/X
                                  314 GET /comments/X/3
                                  261 POST /stories/X/unvote
                                  212 GET /login/2fa
                                  203 POST /stories/X/suggest
                                  191 POST /login/2fa_verify
                                  156 GET /comments/X/4
                                  125 POST /stories/X/unsave
                                   99 POST /comments/X/delete
                                   94 POST /logout
                                   79 GET /comments/X/5
                                   60 POST /stories/X/unhide
                                   45 GET /comments/X/6
                                   30 GET /comments/X/7
                                   15 POST /comments/X/undelete
                                   14 GET /comments/X/8
                                   13 GET /comments/X/9
                                   12 GET /comments/X/3170
                                   12 GET /comments/X/10
                                    9 GET /comments/X/
                              
                              1. 2

                                Yeah, the workload generator is shaping up pretty nicely! FWIW, it looks like the next scaling bottleneck lobste.rs is likely to experience is Ruby at ~100x current load on an 8-core server, followed by MySQL scaling of the transactional update to the traffic stats around ~2000x current load. So still quite a bit of headroom!

                                Also, I made (yet another) stupid with the query above: it now doesn’t include the frontpage, which is arguably quite important, due to a missed |. Try again? O:)

                                grep -vE "assets|fetch_url_attributes|check_url_dupe|set_new_password|reset_password|forgot_password|token" lobsters.log
                                 | sed -e 's/.*\(GET\|POST\)/\1/'
                                       -e 's/\/\(s\|stories\)\/[^\/ ]*/\/stories\/X/'
                                       -e 's/\/u\/[^\/ ]*/\/u\/X/'
                                       -e '/^GET / s/X\/.\+/X/'
                                       -e 's/\/comments\/[^\/ ]*/\/comments\/X/'
                                 | awk '/^(GET|POST)/ {print $1" "$2}'
                                 | grep -E ' /(stories|comments|login|logout|u/|recent|$)'
                                 | sort | uniq -c | sort -rnk1,1
                                
                                1. 1

                                  Nice to hear we should have smooth sailing for some time.

                                   870987 GET /
                                   193915 GET /u/X
                                   105680 GET /comments
                                    27731 GET /comments.rss
                                    23155 GET /recent
                                    18235 POST /comments/X/upvote
                                    13730 POST /stories/X/upvote
                                     9141 POST /comments
                                     7137 GET /login
                                     5572 GET /comments/X/reply
                                     2512 POST /login
                                     2233 GET /comments/X/edit
                                     2055 POST /comments/X
                                     1735 POST /stories/X/save
                                     1574 GET /recent/page/2
                                     1522 POST /stories
                                     1422 POST /stories/X/hide
                                     1159 POST /comments/X/downvote
                                      801 GET /comments/X/2
                                      606 GET /recent/page/3
                                      600 POST /stories/X
                                      391 POST /comments/X/unvote
                                      347 POST /stories/X/downvote
                                      340 GET /comments/X
                                      316 GET /recent/page/4
                                      314 GET /comments/X/3
                                      261 POST /stories/X/unvote
                                      212 GET /login/2fa
                                      203 POST /stories/X/suggest
                                      191 POST /login/2fa_verify
                                      187 GET /recent/page/5
                                      156 GET /comments/X/4
                                      125 POST /stories/X/unsave
                                      125 GET /recent/page/6
                                      110 GET /recent/
                                       99 POST /comments/X/delete
                                       98 GET /recent/page/7
                                       94 POST /logout
                                       83 GET /recent/page/8
                                       79 GET /comments/X/5
                                       73 GET /recent/page/9
                                       60 POST /stories/X/unhide
                                       59 GET /recent/page/10
                                       47 GET /recent/page/11
                                       45 GET /comments/X/6
                                       31 GET /recent/page/13
                                       30 GET /recent/page/22
                                       30 GET /comments/X/7
                                       29 GET /recent/page/14
                                       28 GET /recent/page/33
                                       28 GET /recent/page/21
                                       27 GET /recent/page/28
                                       26 GET /recent/page/12
                                       25 POST /
                                       25 GET /recent/page/24
                                       25 GET /recent/page/23
                                       25 GET /recent/page/18
                                       25 GET /recent/page/17
                                       25 GET /recent/page/115
                                       24 GET /recent/page/40
                                       24 GET /recent/page/27
                                       24 GET /recent/page/19
                                       24 GET /recent/page/15
                                       22 GET /recent/page/80
                                       22 GET /recent/page/60
                                       22 GET /recent/page/16
                                       21 GET /recent/page/37
                                       21 GET /recent/page/25
                                       20 GET /recent/page/95
                                       20 GET /recent/page/31
                                       19 GET /recent/page/96
                                       19 GET /recent/page/39
                                       19 GET /recent/page/30
                                       19 GET /recent/page/112
                                       19 GET /recent/page/104
                                       18 GET /recent/page/35
                                       18 GET /recent/page/20
                                       18 GET /recent/page/134
                                       17 GET /recent/page/42
                                       16 GET /recent/page/67
                                       16 GET /recent/page/124
                                       15 POST /comments/X/undelete
                                       15 GET /recent/page/92
                                       15 GET /recent/page/84
                                       15 GET /recent/page/77
                                       15 GET /recent/page/68
                                       15 GET /recent/page/66
                                       15 GET /recent/page/57
                                       15 GET /recent/page/36
                                       15 GET /recent/page/131
                                       15 GET /recent/page/125
                                       15 GET /recent/page/122
                                       15 GET /recent/page/121
                                       15 GET /recent/page/116
                                       15 GET /recent/page/114
                                       15 GET /recent/page/101
                                       15 GET /recent/page/100
                                       14 GET /recent/page/87
                                       14 GET /recent/page/59
                                       14 GET /recent/page/45
                                       14 GET /comments/X/8
                                       13 GET /recent/page/98
                                       13 GET /recent/page/82
                                       13 GET /recent/page/51
                                       13 GET /recent/page/135
                                       13 GET /recent/page/133
                                       13 GET /recent/page/128
                                       13 GET /recent/page/120
                                       13 GET /recent/page/108
                                       13 GET /comments/X/9
                                       12 GET /recent/page/94
                                       12 GET /recent/page/93
                                       12 GET /recent/page/89
                                       12 GET /recent/page/69
                                       12 GET /recent/page/65
                                       12 GET /recent/page/53
                                       12 GET /recent/page/52
                                       12 GET /recent/page/49
                                       12 GET /recent/page/46
                                       12 GET /recent/page/38
                                       12 GET /recent/page/34
                                       12 GET /recent/page/136
                                       12 GET /recent/page/132
                                       12 GET /recent/page/130
                                       12 GET /recent/page/117
                                       12 GET /recent/page/102
                                       12 GET /recent/page/
                                       12 GET /comments/X/3170
                                       12 GET /comments/X/10
                                       11 GET /recent/page/90
                                       11 GET /recent/page/88
                                       11 GET /recent/page/79
                                       11 GET /recent/page/71
                                       11 GET /recent/page/64
                                       11 GET /recent/page/129
                                       11 GET /recent/page/123
                                       11 GET /recent/page/118
                                       11 GET /recent/page/109
                                       11 GET /recent/page/105
                                       10 GET /recent/page/85
                                       10 GET /recent/page/78
                                       10 GET /recent/page/74
                                       10 GET /recent/page/70
                                       10 GET /recent/page/63
                                       10 GET /recent/page/61
                                       10 GET /recent/page/55
                                       10 GET /recent/page/43
                                       10 GET /recent/page/41
                                       10 GET /recent/page/26
                                       10 GET /recent/page/126
                                       10 GET /recent/page/119
                                  
                                  1. 1

                                    Hehe, yeah, I think you should be good ;)

                                    Hmm, now there’s no entry there for GET /stories/X? Did you perhaps miss the first line when copying? Looks like the data is all from the same logging period though, so I can just manually combine them :) Thanks!

                            2. 1

                              I also just realized that we’re missing the vote distribution for comments, which’d be super handy!

                              SELECT ROUND(counted.cnt, -1) AS bucket,
                                     COUNT(comment_id),
                                     RPAD('', LN(COUNT(comment_id)), '*')
                              FROM (
                                  SELECT comment_id, COUNT(id) AS cnt
                                  FROM votes WHERE comment_id IS NOT NULL
                                  GROUP BY comment_id
                              ) AS counted
                              GROUP BY bucket
                              
                              1. 1
                                +--------+-------------------+--------------------------------------+
                                | bucket | COUNT(comment_id) | RPAD('', LN(COUNT(comment_id)), '*') |
                                +--------+-------------------+--------------------------------------+
                                |      0 |             89907 | ***********                          |
                                |     10 |             27614 | **********                           |
                                |     20 |              2825 | ********                             |
                                |     30 |               637 | ******                               |
                                |     40 |               177 | *****                                |
                                |     50 |                61 | ****                                 |
                                |     60 |                29 | ***                                  |
                                |     70 |                 6 | **                                   |
                                |     80 |                 5 | **                                   |
                                |     90 |                 5 | **                                   |
                                |    100 |                 3 | *                                    |
                                |    150 |                 1 |                                      |
                                +--------+-------------------+--------------------------------------+
                                
                    1. 15

                      Hey folks,

                      Jon messaged me a day or two ago. I gave him the standard answer about these sorts of inquiries: I’m happy to run queries that don’t reveal personal info like IPs, browsing, and voting or create “worst-of” leaderboards celebrating most-downvoted users/comments/stories, that sort of thing. I can’t volunteer to write queries for people, but the schema is on GitHub and the logs are MySQL and nginx, so it’s straightforward to do.

                      A couple years ago jcs ran some queries for me and I wanted to continue that especially as the public stats that answered some popular questions have been gone for a while. It’s useful for transparency and because it’s just fun to investigate interesting questions. I’ve already run a few queries for folks in the chat room (the only I can remember off the top of my head is how many total stories have been submitted; we passed 40k last month).

                      I asked Jon to post publicly about this because it sounded like he had significantly more than one question he was idly curious about, to help spread awareness that I’ll run queries like this, and get the community’s thoughts on his queries and the general policy. I’ll add a note to the about page after this discussion.

                      I’m going offline for a couple hours for a prior commitment before I’ll have a chance to run any of these, but it’ll leave plenty of time for a discussion to get started or folks to think up their own queries to post as comments.

                      1. 3

                        Wasn’t the concept of Differential Privacy developed to allow for exactly the purpose of querying databases containing personal data while maintaining as much privacy as possible? Maybe this could be employed?

                        1. 3

                          In this particular case I don’t think the counts are actually sensitive, so it’s unclear that applying DP is even necessary. But I’ll ping Frank McSherry who’s one of the primary proponents of DP in academia nowadays and see what he thinks :) Maybe with DP we could extract what is arguably more sensitive information (e.g., by reducing the bin widths).

                          1. 1

                            That seems doable, but I have the strong suspicion that if I wing it I’ll screw something up and leak personal info. So hopefully Frank can chime in with some good advice.

                            1. 4

                              I’m here! :D I’m writing up some text in more of a long-form “blog post” format, to try and explain what is what without the constraints of trying to fit everything in-line here. But, some high-level points:

                              1. Operationally queries one and two are pretty easy to pull off with differential privacy (the “how many votes per article” and “how many votes per user” queries). I’ve got some code that does that, and depending on the scale you could even just use it, in principle (or if you only have a SQL interface to the logs, we may need to bang on them).

                              2. The third query is possibly not complicated, depending on my understanding of it. My sed-fu is weak, but to the extent that the query asks only for the counts of pre-enumerable strings (e.g. POST /stories/X/upvote) it should be good. If the query needs to discover what strings are important (e.g. POST /stories/X/*) then there is a bit of a problem. It is still tractable, but perhaps more of a mess than you want to casually wade into.

                              3. Probably the biggest question mark is about the privacy guarantee you want to provide. I understand that you have a relatively spartan privacy policy, which is all good, but clearly you have some interest in doing right by the users with respect to their sensitive data. The most casual privacy guarantee you can give is probably “mask the presence / absence of individual votes/views”, which would be “per log-record privacy”. You might want to provide a stronger guarantee of “mask the presence absence of entire individuals”, which could have a substantial deleterious impact on the analyses; I’m not super-clear on which guarantee you would prefer, or even the best rhetorical path to take to try and discover which one you prefer.

                              Anyhow, I’m typing things up right now and should have a post with example code, data, analyses, etc. pretty soon. At that point, it should be clearer to say “ah, well let’s just do X then” or “I didn’t realize Y; that’s fatal, unfortunately”.

                              EDIT: I’ve put up a preliminary version of a post under the idea that info sooner rather than later is more helpful. I’m afraid I got pretty excited about the first two questions and didn’t really do much about the third. The short version of the post is that one could probably release information that leads to pretty accurate distributional information about the multiplicities of votes, by articles and by users, without all of the binning. That could be handy as (elsewhere in the thread) it looks like binning coarsely kills much of the information. Take a read and I’ll iterate on the clarity of the post too.

                              1. 1

                                To follow up briefly on this: yes, it would be useful to avoid the binning so that we could feed more data to whatever regression we end up using to approximate the underlying distribution.

                          2. 2

                            For those who are curious, I’ve started implementing the workload generator here. It currently mostly does random requests, but once I have the statistics I’ll plug them in and it should generate more representative traffic patterns. It does require a minor [patch](https://github.com/jonhoo/trawler/blob/master/lobsters.diff to the upstream lobste.rs codebase, but that’s mostly to enable automation.

                          1. 1

                            @pushcx seems I missed /recent and /u in that list, which would be a little unfortunate. Here’s a fixed command:

                            grep -vE "assets|fetch_url_attributes|check_url_dupe" lobsters.log
                             | sed -e 's/.*\(GET\|POST\)/\1/'
                                   -e 's//\(s\|stories\)\/[^\/ ]*//stories\/X/'
                                   -e 's//u\/[^\/ ]*//u\/X/'
                                   -e '/^GET / s/X\/.\+/X/'
                               -e 's//comments\/[^\/ ]*//comments\/X/'
                             | awk '/^(GET|POST)/ {print $1" "$2}'
                             | grep -E ' /(stories|comments|login|logout|u/|$)'
                             | sort | uniq -c | sort -rnk1,1
                            
                            1. 4

                              From an sysadm pov, that kind of metrics are very interesting for monitoring purposes (I’d just add the latency with it).

                              1. You can quickly see when there is an issue after a deployment (or something else). For example if the number of upvote/hour is at 0 since your last deployment.
                              2. You can see which routes are the slowest/more used so you can improve them
                              3. See when a new deployment kills the performances
                              4. Correlate with other metrics …

                              In my previous and current jobs we’ve been using Prometheus for that and it’s SUPER useful. Although I recon that it’s probably too huge for just a Lobsters app.

                              1. 1

                                It’s a little bit trickier to extract latency measurements from the nginx logs + the db without additional instrumentation. I agree with you though, those kinds of numbers would be very useful for system monitoring. For my purposes, they’re not particularly interesting, as I’d use the resulting workload generator to generate load and then measure system performance.

                              1. 5

                                It’s 3 AM here so I haven’t thought super hard about the data you’re suggesting retrieving, but offhand it seems okay. I do have one clarifying question; this:

                                It could also be extremely useful to other researchers who may want to build solutions for “real” web applications.

                                seems to imply that the data might be shared. How? Will it be fully public? From what I see here I’m pretty sure I’d be fine with that, I’m just wondering :)

                                Also, as a side note, I assume you’ll submit your paper to Lobste.rs when you’re done ;)

                                Those performance results sound super interesting, checking out the links you posted is on my todo list now.

                                1. 5

                                  I was imagining that the results of running these queries would be posted publicly by @pushcx, or whomever ends up running them. If you or others are concerned about that, I could just use them directly myself, but I think it’d be advantageous to just make them public :)

                                  Hehe, yes, the paper will appear here when it is (hopefully) published.

                                1. 6

                                  Shouldn’t this metadata be public (or, at least available to all lobste.rs members), or not given to any 3rd parties at all? If there’s a reason for it not being public, then there’s risk involved with giving it to 3rd parties. (Who may have best data security ever, but lose data for reasons beyond their control)

                                  Personally I’d love to see stats that could be generated by scraping the site anyway available to everyone, all the time, it’s interesting to see these kinds of distributions and things.

                                  Hit counts which are only available from web server logs seems more sensitive / private and less interesting to me.

                                  1. 4

                                    Oh, sorry, I wasn’t implying that this data needed to be sent to me specifically; I’d be happy for the metrics to be posted somewhere public. I think there is an argument against making the entire db public though:

                                    • Password hashes shouldn’t be public
                                    • User e-mail addresses shouldn’t be public
                                    • Who voted for/against what shouldn’t be public
                                    • “Hidden” lists shouldn’t be public
                                    • Private messages shouldn’t be public

                                    etc., One option would be to anonymize the database dump before posting it, but I think having some well-defined metrics defined instead will get us quite far with less of a potential for accidentally leaking something.

                                    As for scraping the site, I completely agree with you. It would be super useful to get a dump of all stories + comments + vote counts for the purposes of re-constructing (or even just backing up) lobste.rs elsewhere.

                                    In my particular case, hit counts are important for a couple of reasons:

                                    • It shows which pages are more frequently accessed, which in turn tells me how often different queries are accessed.
                                    • If story IDs aren’t masked out, it tells us the popularity distribution of stories (i.e., key skew in database terms).
                                    • It tells us access patterns over time – how quickly does the popularity of particular stories wax and wane.
                                    1. 7

                                      One option would be to anonymize the database dump before posting it

                                      What has to be factored in here is that what’s publicly on the site can be compared to what’s in the dump to deanonymize quite a bit of users. That a bit easier given a relatively-small number of commenters and submitters whose votes are already visible on main site.

                                      1. 3

                                        Related to public backups of the site, a few months ago Archive Team scraped the entire site. I don’t know where to find the archive they made (maybe it all just went into the Wayback Machine?) or if they’re continuing to pick up new stories.

                                    1. 1

                                      It doesn’t allow people to customize the UI or turn off tracking does it?

                                      1. 2

                                        Fwiw you can rearrange the ui elements in the browser and write custom CSS to style it if you really want to get deep into things.

                                        As for manually injecting elements into the XUL interface, AFAIK your only option is using an addon to hook into the webextensions api.

                                        1. 1

                                          My understanding is that there is no more XUL interface, or at least that they’re removing access to it with the intent to remove it entirely soon.

                                          1. 2

                                            The chrome is still XUL - try debugging the browser with the browser toolbox and you’ll see the XUL elements that make up UI components. afaict they’re just removing the capability for extensions to modify the ui directly.

                                          2. 1

                                            Related to this, do you know of a way (using custom CSS) to move the tab bar to the bottom of the window? That’s the feature I’d miss the most from Vivaldi/Opera/older Firefoxes…

                                        1. 1

                                          I have attempted to write up a comment summarizing the discussion, as well as proposing next steps at https://github.com/jcs/lobsters/issues/376#issuecomment-315805596. Feedback welcome. /cc @pushcx @mcheely @angersock @peter @nickpsecurity @alynpost @orib @hwayne etc.

                                          1. 2

                                            I think the problem with this is that everyone defines hostile and abusive differently. There would have to be a clear definition of what this is. It might also help if it was possible to show voting histories of people so there is more transparency and over sight.

                                            1. 1

                                              I agree. See also the discussion of better alternatives further up.

                                            1. 6

                                              Today I learned that downvoting a comment requires you to pick a category. I guess I’ve never tried to do that here before.

                                              The comment provided as an example is the first comment I tried to downvote here on Lobste.rs…

                                              And indeed, none of the available categories fit.

                                              If there was a ‘destructive’ category, I’d pick that one. Meanwhile, I’ll use ‘troll’, which is clearly not correct. AFAIK.

                                              [EDIT: uh-oh, it looks as if I’ve committed a “me-too”!]

                                              1. 5

                                                I only learned that downvotes require a category today as well. And I was pleasantly surprised both because I realized that I haven’t needed to downvote a comment here before, and because I really like that lobste.rs requires a reason for a downvote. I feel like your comment had value beyond just a me-too, so you’re fine :)

                                                1. 6

                                                  Interestingly, gave me an opportunity to upvote that comment. Don’t know why someone’s personal feelings, which is what they are describing there should be less valid because of the color of their skin. I thought that was what we were all striving for.

                                                  1. 4

                                                    People reading that comment who missed the thread might not know it was very context-sensitive. Remember that the context (OP) is specific people pushing a specific set of political views on everyone asking that all disagreement be censored. They say they benefit minorities but wont allow them to have a say if they have different beliefs. Coraline et al are uncompromising in that the options are (a) agree with them pushing same things or (b) shut up and leave every public space they take over.

                                                    With that backdrop, I read the various Github articles and the OP. She constantly talked about extreme negative reactions she got as if it’s incidental to being a minority. She was a minority, did some work, and piles of hate emerged. She never mentions when doing so that she aggresively evangelizes, insults, and coerces FOSS projects usually with a pile of likeminded people behind her. I kept bringing that behavior up since I think her showing up at people’s doorsteps insulting them and telling them to do her bidding in their projects might be why people dont like her. That pisses all types of people off here in the Mid-South, including minorities. Consistently. I imagine some in other areas, too.

                                                    Anyway, in the thread you linked, my main comment on that article was judged by site as follows:

                                                    +73 yes -4 incorrect -1 off-topic -8 troll

                                                    It means the main post got overwhelming support esp considering how few upvotes I normally get. The others were peripheral supporting it as part of a larger debate. Anyone trying to judge the linked one should probably look at OP and first comment to get context:

                                                    https://lobste.rs/s/js3pbv/antisocial_coding_my_year_at_github#c_h8znxo

                                                    Im just a political moderate calling out hypocrisy/deceit of an article’s source (i.e. source integrity) and protecting right to dissent as usual. I do it on all topics. Even my favorites on occasion. On political ones, people tend to have strong emotional reactions that clouds judgment or just evokes strong reactions. Im not saying whose right or wrong so much as disagreement they take personally, get disgusted/angry, and will hit any button to make that person or post disappear.

                                                    I think I warned of that in either linked thread or Community Standards discussion. Both then and now, people started calling out others that should disappear with often opposite views of what should be allowed. There was no consensus except against comments that are blatantly harmful where there is a consensus by most peeple that it’s abusive. The same thing I see play out in person every day. So, I oppose comment deletions or bans in political situations without consensus so long as people keep it civil and about specific claims with supporting evidence. And if one side can speak, the other parties better be able to as well.

                                                    And a minimum of politics on Lobsters period! Keep it focused on tech and such. Somone had to post something by a decietful activist on politics pushing a mix of truth and propaganda. And that hit my mental button of calling them out staying as civil and factual as I could despite knowing with every word I might be censored for it. Might. The upvotes from my first comment were reason I kept taking the risk of more argument given there was a surge of dissent that needed to be represented. Not just me. I always help the underdogs. :)

                                                    Note: That was long as we were just talking about but I wanted context and intent clear given it’s about whether to filter or ban me. I also hold no grudges against anyone who did. It’s their deeply-held, personal beliefs about what’s right and wrong. People will do what you believe is necessary there.

                                                    Note 2: Lunch break is over. Darn. I was hoping for tech over politics. Ill do what’s necessary, though, since I value and respect this community. Gotta defend dissent as it’s critical.

                                                    1. 3

                                                      While I disagree with your positions on the topic of the OP, that’s not really what I wanted to bring attention to in this thread. And, as you correctly point out, the longer post you had there does contribute to the discussion. This is why I specifically linked only to that one comment, because that is the only one I feel is not contributing, constructive, or otherwise meaningful as a part of the larger thread. Under no circumstances do I think any of what happened in that thread is cause for banning or deletion; on that, we are in complete agreement. What I wanted to highlight in this topic is that we should have a way of discouraging comments that are solely inflammatory without carrying other value, and I believe that particular one was of that kind. I did not downvote your other posts despite disagreeing with them, because (as also mentioned elsewhere in this thread), I do not think disagreement should be a reason for downvoting. We can have a whole different discussion about how politics and tech mix, but that does not belong in this thread.

                                                      1. 3

                                                        This is why I specifically linked only to that one comment, because that is the only one I feel is not contributing, constructive, or otherwise meaningful as a part of the larger thread. Under no circumstances do I think any of what happened in that thread is cause for banning or deletion; on that, we are in complete agreement.

                                                        Well, my respect just went up for you quite a bit. Very reasonable position far as critiques go. The selected comment was lower info than the other one and maybe even unnecessary. Likely because it was part of a back and forth on politics where comment quality on all sides (including my own) tend to get lower as it goes on. One of reasons I don’t like political discussions in low-noise sites like Lobsters. They also can have less info since more of the specific points and context is already defined where the replies start just implying that stuff with less info content in general. That one was some combination of those.

                                                        In any case, I appreciate you clarifying your position. I at least get why you’d want to see less of that kind of comment than the main one.

                                                        1. 4

                                                          I’m glad we’ve found common ground. As I’ve said elsewhere (this thread is getting pretty large), I don’t want to see downvotes used as a way to signal disagreement, nor do I want them to be used to “punish” a particular user or otherwise label the user as bad. Downvotes to me are a way of signaling that a particular comment is unwanted, along with the reason why, nothing more, nothing less.

                                                          1. 1

                                                            I’m fine with that as long as there’s a consensus across majority of community’s users. That’s really all I ask with these sorts of things even though I’m biased toward free speech or low censorship. Your proposal isn’t a big risk to that esp given it’s mostly a tech-focused forum.

                                                    2. 1

                                                      alynpost’s suggestion meets my needs and I withdraw my support for a new category.

                                                    1. 65

                                                      I feel like the last few weeks have had a rash of sneering, rude, or hostile comments. Even a very little of that tells every reader this isn’t a friendly place to collaborate and learn things. I’m not going to link to those comments, I’m going to link to the good stuff from just the last 24 hours:

                                                      I want to see people excited for learning, admitting ignorance, thinking it’s worth their time to deep dive, sharing resources, building context, basically this whole thread.

                                                      This serious conversation where collaborators think it’s worthwhile to give and feel it’s safe to ask, it takes vulnerability and care. Scorn, showing off, and put-downs shut people down fast, especially the newbies and minorities. Kindness is the fuel of a good community.

                                                      I would like this downvote category to be added.

                                                      1. 17

                                                        I feel like I’ve been seeing the same thing, and I’ve been trying to come up with a consistent way to categorize it. I think what bothers me mostly boils down to seeing posts that are primarily nonconstructive anger/hostility. It’s the posts that don’t do anything except complain about something someone did, or the state of some tech they don’t like, or the way some project is run. Whether it’s bitching that systemd isn’t unix or that a trans woman felt betrayed by her employer, if your only idea is destructive (e.g. this project is useless, this person shouldn’t be employed, this code shouldn’t exist, everything in web-dev is crap) I don’t really want to hear about it, and I don’t think it contributes to the quality of discussion on the site. There are perhaps a few other things that don’t fall into that category, but that seems to capture a lot of it.

                                                        I’m not sure if that’s the basis for a useful construct for categorization though, as it may be possible to re-structure such comments in a way that seem constructive on their face. Colloquially speaking I’d like a place to discuss technology with significantly less bellyaching and a lot more goodwill.

                                                        1. 9

                                                          Regrettably, not all comments can be constructive and positive, “yes and” tricks notwithstanding.

                                                          I think that it’s sufficient to use the tools we already have, especially if the alternative is a nebulous “hostile” flag that is basically going to be used to silence viewpoints folks don’t agree with.

                                                          Using your own example, right, it’s trivially easy to just paint a disparaging brush over criticisms about, say, “a trans woman feeling betrayed” when the context surrounding it is actually kinda interesting–not that the article should’ve been here in the first place, mind you, but here we are.

                                                          That kneejerk reaction and othering to oversimplify the opposition’s points is not as constructive as saying, say, “@nickpsecurity you have some valid points perhaps but jesus christ write shorter posts”.

                                                          1. 17

                                                            I think this is an uncharitable interpretation of what mcheely, myself and others are saying in this thread. We’re not (well, at least I’m not; I can’t speak for everyone) arguing that all comments need to be positive — dissent and disagreement is vital to good discussion — but rather that when a comment is negative, it should aim to be constructive, non-hostile, and tolerant. I don’t see how any of those would stifle dissenting voices, nor remove the context that you correctly point out is so vital. Nor am I arguing that posts should be penalized for being complex or long. In fact, the specific example I highlighted is very short; as pointed out elsewhere, it’s only that one comment that I feel like deserves a downvote.

                                                            1. 12

                                                              I think this is an uncharitable interpretation

                                                              I’d like a place to discuss technology with significantly less bellyaching and a lot more goodwill

                                                              I think uncharitable should be a new downvote flag. I’m okay with dissent, I’m okay with people being assholes, I’m even somewhat okay with outright negativity, as long as the comment is also charitable to the other viewpoint. Because nothing makes my blood boil like comments (online or face to face) that outright misinterpret or ignore what someone else is saying. It always feels like it’s done out of malicious dishonesty, arrogance, and spite, all at the same time. Uncharitable arguments tend to make me leave a discussion immediately with an intense desire to strangle someone.

                                                              I believe everyone on this site has the intellectual capacity to be charitable in their comments. And if someone consistently refuses to do so, I think they should be banned. Not as a punishment, but to keep a strong community of healthy communicators. The uncharitable downvote itself would serve as a reminder to be charitable before you vote, and perhaps make us all a little more charitable to each other.


                                                              And now I’m going to be salty. So if you get the point then feel free to stop reading, because the above 2 paragraphs were by far the most important.

                                                              Being directly uncharitable to another commenter’s point of view is obviously frustrating, but so are uncharitable top level comments. Lets take systemd for example, because for some reason no one can discuss the topic charitably. If you want to discuss specific disagreements with some aspect of systemd, be my guest. But if you say, e.g., Lennart Pottering is a moron, or systemd is terrible software, or complain about how systemd is killing Linux, then

                                                              <deeply-aggressive-language>

                                                              you should shut the hell up and meditate on why you have nothing better to do than express such mundane and idiotic generalizations.

                                                              </deeply-aggressive-language>

                                                              Is that uncharitable to say? Maybe. It’s hard to be charitable to uncharitable comments. I did my best to speak against expressing negative generalizations without addressing anything specific. Generalizing is the easiest way to be uncharitable, because a negative generalization automatically dismisses anything positive. If some people can’t find the time to phrase something charitably, then I want a way to tell them that I expect better. I expect some fucking intellect.

                                                              Let’s examine how these each are uncharitable, regardless of whether you or I agree:

                                                              “Lennart Pottering is a moron.” Ok first, he clearly isn’t. If you disagree with his design decisions that does not automatically make you smarter than him. I would challenge you to do better. If you legitimately could then you have a lot more value to contribute to the discussion than personal attacks, otherwise you’re an arrogant child who deserves downvotes for wasting everyone’s time.

                                                              “Systemd is terrible software.” This is a legitimate opinion, but the expression has all the same same problems as insulting Lennart Pottering. To comment, you should enough relevant knowledge that you can frame your opinion charitably and legitimately discuss some aspect of systemd. If you don’t you should post a question at most, or at least refrain from being an asshole when expressing your opinions.

                                                              “Systemd is killing Linux.” This is my key example. It’s also a legitimate opinion, but it’s still an opinion, and one that’s terribly hard to prove. By expressing such an unprovable (and in this case polarized) opinion, you’re being uncharitable to the reader. There is no way to argue with you, the point is simultaneously undefendable and undefeatable. The reader hasn’t even tried to talk to you and you are already being preemptively uncharitable to their very thoughts. At best you provide the root of a comment tree that isn’t worth reading, and can be collapsed in one click. And for everyone who agrees with you, you’ve inflicted your negative emotions on them, trapped them into feeling uncharitable as well until they can distract themselves from the subject.

                                                              I never click on systemd threads anymore, because they’re so predictably awful. And it all comes down to people being uncharitable. I don’t want that in my life. And it’s sad, because I actually would like to discuss systemd. Lennart Pottering probably feels the same. Can you blame him?

                                                              1. 6

                                                                Thanks for these examples, they are top! It’s these kinds of unsupported and unsupportable statements that get me railed up frequently.

                                                                1. -1

                                                                  “Lennart Pottering is a moron.” Ok first, he clearly isn’t. If you disagree with his design decisions that does not automatically make you smarter than him. I would challenge you to do better. If you legitimately could then you have a lot more value to contribute to the discussion than personal attacks

                                                                  I’d say that to do nothing is to do better than aggressively pushing software that acts as a DDoS relay, breaks sound output, and/or renders systems unbootable - and then pushing other OSS applications to break compatibility with OSes that don’t adopt your buggy software.

                                                                  The reader hasn’t even tried to talk to you and you are already being preemptively uncharitable to their very thoughts. At best you provide the root of a comment tree that isn’t worth reading, and can be collapsed in one click. And for everyone who agrees with you, you’ve inflicted your negative emotions on them, trapped them into feeling uncharitable as well until they can distract themselves from the subject.

                                                                  So what’s the constructive thing to do here? If other people want to run Pottering software that’s their lookout - as far as I’m concerned he’s had his three strikes on my systems. But how do I stop the creeping hard-dependency making its way through the ecosystem?

                                                                  What would it take to get program maintainers to stop adopting systemd-isms? At least “systemd is killing linux” gives them one datapoint, maybe a reason to think twice. I’m open to anything more positive I could do in the cause of keeping running my programs and not having updates break my computer (which really doesn’t seem like so much to ask).

                                                                  it’s sad, because I actually would like to discuss systemd. Lennart Pottering probably feels the same. Can you blame him?

                                                                  Yes I can. I never wanted systemd, never asked for it, but it still broke my computer.

                                                                  1. 3

                                                                    So what’s the constructive thing to do here?

                                                                    I can think of a lot of potentially constructive things to do:

                                                                    • Engage in reasonable discussions with project maintainers about possible alternatives to hard dependencies.
                                                                    • Fork a project to remove systemd dependencies
                                                                    • Use a distro that doesn’t rely on systemd
                                                                    • When posting on lobsters about systemd try something like “If you’re not a fan of systemd’s design, check out fooproject from bardude. It does x, y, and z better, and refrains from p and q. It’s optionally available as part of the wankelrotary distribution, and I’ve found it works pretty well, except that the only font color I can pick in gnome is dark purple. I’m willing to live with that.”
                                                                    • If you don’t have the experience or information to do any of those things, start building it.

                                                                    What would it take to get program maintainers to stop adopting systemd-isms? At least “systemd is killing linux” gives them one datapoint

                                                                    That might be true, but I would argue that those maintainers probably have dedicated forums for that sort of feedback, making re-hashing it here unnecessary. It may be that you’ve said your piece in those forums and the maintainers have opted to adopt systemd anyway. If that’s the case, I don’t think re-hashing the arguments here is going to change their mind, and I agree with @peter that doing so detracts from the quality of discussion and discourages the participation of folks who do have constructive ideas to contribute.

                                                                    1. 1

                                                                      Fork a project to remove systemd dependencies

                                                                      Use a distro that doesn’t rely on systemd

                                                                      Done/doing, but it feels like holding back the tide. I can’t fork everything I use.

                                                                      When posting on lobsters about systemd try something like “If you’re not a fan of systemd’s design, check out fooproject from bardude. It does x, y, and z better, and refrains from p and q. It’s optionally available as part of the wankelrotary distribution, and I’ve found it works pretty well, except that the only font color I can pick in gnome is dark purple. I’m willing to live with that.”

                                                                      But it’s not about having something else to do the things, because the things aren’t useful in the first place. It’s not users who are pushing systemd, it’s distributions, and while Debian at least had the discussion loundly and publicly, I don’t see any realistic chance of moving them away from systemd now.

                                                                    2. 3

                                                                      Systemd wasn’t the point of my comment at all, but you have demonstrated that I picked a good hot button example.

                                                                      I’d say that to do nothing is to do better than aggressively pushing software…

                                                                      Sure. But that doesn’t make him a moron. And does he even aggressively push the software? Did he personally direct the majority of distros to adopt systemd? As far as I know he just developed systemd for Red Hat, and other people jumped on board. I particularly remember Arch Linux was excited to move to systemd, proud that they were so ahead of the curve. But now people uncharitably retcon history, pretending Arch was coerced to make the move by the changing ecosystem. It’s fine to criticize his choices, it’s uncharitable to attack his intelligence, and it’s uncharitable to rewrite history against him.

                                                                      as far as I’m concerned he’s had his three strikes on my systems

                                                                      This is a alright response—I certainly wouldn’t call this uncharitable. If you don’t like the software personally on your systems, that’s your prerogative.

                                                                      But how do I stop the creeping hard-dependency making its way through the ecosystem? What would it take to get program maintainers to stop adopting systemd-isms?

                                                                      Good question. I have a few ideas but they’re beyond the scope of what I’d like to talk about. But I guarantee you making uncharitable attacks against the developers will not help. It just prevents your feedback from being heard, and I think it’s fairly intuitive why. There are many posts from OSS maintainers describing their growing resentment with their community, just from a small minority of uncharitable people. I think anyone who doesn’t like systemd is pretty boned for the foreseeable future, because the community has poisoned its ability to sound rational when criticizing systemd. Perhaps permanently.

                                                                      At least “systemd is killing linux” gives them one datapoint, maybe a reason to think twice.

                                                                      No. It gives them one more reason to think the community is nothing but assholes who aren’t worth interacting with.

                                                                      I’m open to anything more positive I could do

                                                                      Be charitable. Encourage others to be charitable and stop poisoning the developers against the OSS community.

                                                                      Beyond that, voice your use of alternatives. Spread the word. Lead by example. If you just bitch about systemd, you’re teaching anyone around you that there is no better option than to whine and be miserable. Instead I encourage anyone who doesn’t like systemd to consider runit (which I find delightfully simple), or consider BSD if they’re up for a bigger change. Otherwise you kinda have to accept that people aren’t going to build whatever software best suits you because you asked for it. You don’t even get that for software you pay for.

                                                                      Can you blame him? Yes I can.

                                                                      Now this is certainly uncharitable. I asked if you could blame him for not wanting to interact with a community that vilifies him. And you blatantly respond by vilifying him further. If you weren’t just uncharitably ignoring what I was actually asking, and you truly blame him for not appreciating people who hate him, then I must ask you why you hold him to such a high empathetic standard while shamelessly holding yourself at such a low one? However based on your subsequent sentence, it seems far more likely that you just ignored my actual question.

                                                                      1. 0

                                                                        does he even aggressively push the software? Did he personally direct the majority of distros to adopt systemd? As far as I know he just developed systemd for Red Hat, and other people jumped on board.

                                                                        I might believe that if it were the first time this happened, but it isn’t; his projects show this pattern, and other people’s don’t.

                                                                        There are many posts from OSS maintainers describing their growing resentment with their community, just from a small minority of uncharitable people.

                                                                        If resentment is what it takes to get the systemd pushers to go away, I’m ok with that. Like, I don’t want to upset people, but I’d take upset people and working software over the converse.

                                                                        Now this is certainly uncharitable. I asked if you could blame him for not wanting to interact with a community that vilifies him. And you blatantly respond by vilifying him further. If you weren’t just uncharitably ignoring what I was actually asking…

                                                                        So much for your charitability. If that was what you meant to ask then you weren’t clear about it; certainly it wasn’t the question I understood you to be asking.

                                                                        No, I can’t blame him for not wanting to interact with the OSS community; on the contrary, if he would stop doing so I would have no further issue with him.

                                                                        1. 2

                                                                          If resentment is what it takes to get the systemd pushers to go away, I’m ok with that.

                                                                          80% of my post was describing the opposite of that interpretation.

                                                                          If that was what you meant to ask then you weren’t clear about it

                                                                          I said “it’s sad, because I actually would like to discuss systemd. Lennart Pottering probably feels the same. Can you blame him?” You quoted that exact piece of text, and responded to it. How could I have better phrased it for you to understand what I was asking?

                                                                          So much for your charitability.

                                                                          As I said before, it’s difficult to be charitable to uncharitable people. In each of my responses to you I’ve felt the need to question why you’ve uncharitably interpreted something I said.

                                                                          1. 1

                                                                            I said “it’s sad, because I actually would like to discuss systemd. Lennart Pottering probably feels the same. Can you blame him?” You quoted that exact piece of text, and responded to it.

                                                                            Yes I did. I understood that exact piece of text to say something quite different from “I asked if you could blame him for not wanting to interact with a community that vilifies him.” Rereading it I still think it says something quite different from that.

                                                                            Were you asking: “can you blame him for wanting to discuss systemd?” (which I think is the obvious interpretation of your line, and what I initially responded to), “can you blame him for feeling sad?” (not an interpretation I considered), or something else (as your response suggests)?

                                                                            As I said before, it’s difficult to be charitable to uncharitable people. In each of my responses to you I’ve felt the need to question why you’ve uncharitably interpreted something I said.

                                                                            Again that cuts both ways. Throwing around accusations of uncharitability is itself uncharitable, and I think your very first bringing up of systemd fails the charitability bar.

                                                                  2. 1

                                                                    I couldn’t be arsed to read through those threads, as it was obviously going to be a shitstorm from which nothing good can emerge.

                                                                    I read the one comment without context.

                                                                    Someone has an open mind for unpopular thoughts and considers all discrimination equal.

                                                                    Some other examples have been proper shitposts, but this one really isn’t. It’s just unpopular.

                                                                  3. 11

                                                                    I like nickp’s long posts :)

                                                                    1. 6

                                                                      Appreciate it. I gotta figure out a balance. I think those that are long for references are justifiable for discovery process. Opionion write-ups are where I could improve the most.

                                                                      1. 7

                                                                        Agreed. Especially when you go into long historical stuff the walls-of-text are justified–other more opiniony stuff could probably stand to be bullet-points and hyperlinks to relevant background material.

                                                                        Thanks for thinking on it though. :)

                                                                        1. 3

                                                                          I find it helps to reread my opinion posts, identify the key ideas, and find a way to bubble them all to the top. That way the reader knows up front if they want to continue reading. It seems to me to be the right thing to do in almost every type of technical or intellectual communication.

                                                                          For example, when teaching, putting the main idea up front lets the student know what they’re supposed to be learning. Do it the other way, and they have to try to buffer everything you’re saying indiscriminately and hope to tie it together at the end.

                                                                          1. 2

                                                                            I have noticed I put too much info in front of the main point sometimes when it wasn’t strictly necessary to do so. Part of the reason is I treat this as an informal forum where I’m just talking to people free-style. Possibly adding a bit of formality or structure to it using something like you describe could bring them more value. So, yeah, that good advice.

                                                                      2. 3

                                                                        Lmao. I get that a lot. Yeah, I should take another stab at concise writing.

                                                                        1. 2

                                                                          Regrettably, not all comments can be constructive and positive

                                                                          I don’t have the bandwidth for a longer reply, but this assertion is going to need some backing up. Simply asserting it’s impossible to be constructive does not make it true.

                                                                      3. 7

                                                                        I appreciate people being willing to voice unpopular opinions that can be taken as hostile. I don’t like a community of people who tiptoe around my statements, and aren’t willing to tell me that I’m saying something stupid when I say stupid things. I don’t like communities where people feel the need to paste on a fake smile while talking to me, and avoid saying anything confrontational.

                                                                        If there is a category that’s added, I’d want it to be specifically for deliberate distortions, misinterpretation, and blowing statements out of proportion. Possibly call it, “Bad faith”. But if that is added, I think it should be a superset of troll. If it’s added, troll should be removed.

                                                                        1. 2

                                                                          I completely agree that it’s important not to silence opinions just because they are unfavorable, dissenting, or disagreeable. And I think the amount of discussion that’s been going on in this thread shows that most other users of lobste.rs also care about this. It is precisely for the reason you state that I think it’s vital to pick the right word for such a category. It needs to be something that doesn’t convey merely “I disagree” or “I found this offensive”, but rather “this content has no place in the comment section of this story, and is only inflammatory/hostile/unconstructive”.

                                                                          While “troll” captures some of that sentiment, I think there is room for two different categories here (though there may also be a word that captures both without also pulling in too much else): “troll” to me indicates a comment intended to get a rise out of others (although alynpost had a good and wider definition further down), whereas I want this new category to cover statements that are written in such a way that they are either unproductive or hostile. It’s not so much about the content of the comment as much as it is the tone (a comment without content is more likely to be marked as “troll”). The distinction is definitely tricky though, and I don’t claim to have any answers here.

                                                                          1. 1

                                                                            unproductive or hostile

                                                                            Some ideas deserve hostility, and one person’s idea of productive may not be shared with another.

                                                                            Again, I think that the goal is to indicate that a comment is made in bad faith. That is to say, a comment is not written with the intent of discussing a genuinely held belief of the author, nor to expore new ideas, nor to discuss an important weakness in a previous comment. Instead, the comment may be made in order to derail discussion, to make people angry, or to belittle others, or to stroke the poster’s ego.

                                                                            Trolling is one instance of this category of comment, but it seems like this would cover most of the other instances as well.

                                                                            Finally, as someone mentioned, turning the downvote into a process, where the reasons for the downvote are described in detail may be a great idea.

                                                                            1. 3

                                                                              “Some ideas deserve hostility” — while I agree with you in the very general case, I think it’s rare for ideas expressed in comments on this site to deserve hostility. If you can’t make your argument in a civil and constructive way, then I strongly suspect your argument is not particularly strong. You are of course right that two people’s definition of “productive” can vary widely, which is why “unproductive” is probably a poor name for this theoretical tag. However, there are some pretty good other suggestions above that I think do a much better job of communicating that this is much more about tone than it is about content. Take “uncharitable” for example — I believe it fits your description of “bad faith” pretty well, and yet many comments that I would not consider “trolling” could still be considered “uncharitable”. A long rant that clearly took the person a while to write, and which they have put thought into, is likely not trolling (at least I wouldn’t think about it that way), but could certainly be “uncharitable” in the sense of its tone being unnecessarily inflammatory/aggressive/disrespectful.

                                                                        2. 4

                                                                          I closed the GitHub issue following alynpost’s comment at https://lobste.rs/s/wei3a9/downvote_category_for_inappropriate#c_pryqvi. Do you agree with their reasoning that “troll” may be appropriate after all, or do you think we should re-open the GH issue?

                                                                          1. 15

                                                                            I’m using “troll” in the meantime, but I would like to see an explicit “unkind”, “abusive”, “uncharitable”, “jackass”, something.

                                                                            1. 3

                                                                              Some thoughts from a side-discussion with alynpost that I thought I’d share: I like the sentiment expressed by “not even wrong”, but I worry that it’s a tag that can fairly easily be misunderstood or misused. “pointless” is a decent approximation, but unfortunately also has hostile associations. Perhaps “unnecessary” is better? Alternatively, “not germane” and “wanton” are both somewhat accurate, but also pretty obscure.

                                                                              1. 1

                                                                                Maybe “gratuitous”?

                                                                              2. 2

                                                                                My vote is for shenanigans. Mostly because I like the word and thing it would fit very well in this situation.

                                                                                1. 2

                                                                                  I’ll contribute ‘profane’ and ‘taboo’ to your word salad.

                                                                                  1. 5

                                                                                    I’ll proffer ‘snark’.

                                                                                    1. 3

                                                                                      snarky comments can be “good” if they aren’t overly offensive. sometimes they are a good tool to show another point of view.

                                                                                    2. 4

                                                                                      Taboo is an interesting choice. Are some topics taboo? (Yes, I think so.) Will voters be willing to specify taboo as a reason? I wonder if having the option available would cause voters to reconsider.

                                                                                      1. 5

                                                                                        The word taboo in the context of moderation was introduced to me via the article It’s Time For a Class Action Against Facebook to Force Remedies The Government Has Not. I’ll quote the relevant passage. It’s not entirely, well, germane, to how lobste.rs would use the label, but it’s a good survey of the space:

                                                                                        In user interface similar to the selection of other ‘interests’ Provide for the user-selected option of participating in discourse on Sensitive Topics (“Taboos”), and limit the distribution of all user-created ‘Posts’ or their equivalent, to those who have selected to participate in that same Taboo.

                                                                                        • Absolutely Prohibited Taboos: Believable threats of interpersonal physical or sexual harm. Pedophilia.
                                                                                        • State Regulated Taboos: Criticism of the state. Criticism of Political Individuals.
                                                                                        • Religious Taboos: Advocacy of or criticism of any major world religion.
                                                                                        • Political Taboos: Advocacy of or criticism of any political ideology, philosophy, or party. Tolerance for advocacy or criticism of
                                                                                        • Psychological Taboos: Race, Gender, Disability
                                                                                        • Sexual Taboos: Erotica
                                                                                        • Purity Taboos: Gore.
                                                                                        • Honesty Taboos: Ridicule, Shaming, Rallying;
                                                                                        1. 3

                                                                                          I like the honesty of this list. It says what the US Supreme court has said for a long time. You can’t throw out the bath water without throwing out the baby. They, of course, said it better: 1 2 and so on.

                                                                                          Now, online fora are not covered by free speech laws but whenever you govern speech you find the conflict you find in this list - fortunately lobste.rs is not our sole outlet for political views. In fact lobste.rs is not an outlet for political views at all, so how did we get here?

                                                                                          What we need is a personalizable, trainable spam filter that will allows us to auto-hide all this junk and just let us learn about the high drama in the world of algebraic data types.

                                                                                2. 4

                                                                                  Which word would you use to describe the category? I’ve seen these proposed:

                                                                                  • Abusive
                                                                                  • Destructive
                                                                                  • Hostile
                                                                                  • Inappropriate
                                                                                  • *sigh*
                                                                                  1. 2

                                                                                    “Inappropriate” sounds like “off-topic” to me, but any of the rest or these work for me.

                                                                                  2. 1

                                                                                    Comments that fit that description should be straight up deleted/filtered. If deletion isn’t kosher here, maybe a moderator powered “-∞” vote that also prevents further notifications? People would probably take it seriously if it reset their karma to zero (internet points!).

                                                                                    1. 7

                                                                                      You’re letting the vocal mobs win in that case. Better to drop them to bottom of thread hidden by default with a plus or profile setting that can display them. That keeps them out of sight with votes but avoids total censorship where people can still read what was said.

                                                                                      1. 4

                                                                                        That seems excessive. I think we want to be very careful about this kind of downvote which could easily be misused. It’s also possible that a user makes one comment like this, but has other valuable comments and contributions, and the one bad should probably not undo all the good.

                                                                                    1. 3

                                                                                      Based on several of the comments in this thread, I think my initial (pessimistic) response to this proposal may have been valid; “inappropriate”, “hostile”, and “abusive” are all dogwhistles for various forms of moralizing censorship.

                                                                                      At face value, all of these words have a negative connotation, so the obvious response is “yes, I too dislike bad things and we should get rid of them”. Unfortunately, when people actually go to use these downvote buttons, they’re going to unify the more general concept that we can all agree is bad with something much more specific that we won’t. In particular, “innapropriate” is literally just a stand-in for “violates my social conventions”, which could be anything from blaspheming the prophet to speaking out of turn. “Hostile” often just means “disagrees with me emphatically enough” or “has an opinion which could make someone feel bad about themselves”. I’ve seen “abusive” abused relentlessly over the last few years, mostly with similar connotations to “hostile”. Other dogwhistle generic-negative words include “violence”, “hate”, “problematic”, “disrespect”, etc.

                                                                                      We should be extremely careful in allowing these words to become acceptable rationalizations for policing comments. Whether you’re blaspheming against a god or using “her” instead of “xer”, explicit moral judgments shouldn’t be the basis for downvotes. All the other categories are at least somewhat objective.

                                                                                      1. 1

                                                                                        I agree that all the words used in the title are overly pessimistic and tied to subjective opinion. The goal of this tag should not at all be to mark comments that you disagree with, or that you personally find offsensive. There’s some good discussion near the top with alternatives that I think do a better job of communicating that it’s about the tone of the comment more so than its content. Someone also made the observation that it’s likely hard to be “politely hostile toward a group and simultaneously provide actual constructive criticism”, which I think is a good point: if we require comments to be “civil”, better and more productive discussion is likely to ensue without there having to be any kind of censorship of content.

                                                                                      1. 3

                                                                                        while there may be a bit more .. bad? .. comments recently, i think it can be handled either with the current voting system or just by ignoring them. maybe we just need a “hide” flag for comments.

                                                                                        and: if someone says something vaguely offensive, maybe it’s just a communication issue? talk to each other, please. just “be nice”. and listen.

                                                                                        1. 2

                                                                                          I completely agree that the site (and the world for that matter) would be a better place if people were nice and listened. And it’s true that the current voting scheme works pretty well, and that in the majority of cases you can simply choose to ignore comments that you disagree with. However, I think the community should strive to improve where it can, and I believe there is still a fraction of comments that fall through the cracks in the current scheme: comments that are written in an overly aggressive, hostile, unconstructive, and/or inflammatory tone. As pointed out in the OP, these are not well covered by existing tags (“troll” comes closest, but isn’t really appropriate), and I believe the community would be better off if something like that were to be added.

                                                                                        1. 3

                                                                                          It seems to me it is working as intended. You have a knee jerk reaction to somebody you disagree with, try to mark it as troll or incorrect, and feel guilty knowing that it is neither.

                                                                                          1. 2

                                                                                            I’m not sure where you get the impression that any of this is about knee-jerk reactions? As evidenced by this thread, this is clearly something I spent some time debating (both with myself and others). If it were a knee-jerk reaction to something I disagreed with, I personally wouldn’t downvote (as stated elsewhere, I don’t believe that’s what downvotes are for), but I certainly wouldn’t spend time and effort on trying to improve the community by suggesting changes to how downvotes of this particular nature should be handled.

                                                                                          1. 8

                                                                                            There’s only one rule that I know of, babies—God damn it, you’ve got to be kind.

                                                                                            In honor of Kurt Vonnegut, I propose we add an “unkind” tag. It feels like it encompasses the types of interactions that aren’t constructive.

                                                                                            1. 3

                                                                                              “unkind” is a good sentiment, though I worry that “unkind” could also be applied to content that shouldn’t necessarily be downvoted. :/

                                                                                              1. 5

                                                                                                If someone wants to downvote something for a reason that isn’t “accepted,” then they’re not going to give up because the exact sentiment they feel isn’t in the tag list. Already, I assume the default is gonna be “troll” for those situations.

                                                                                                If we want this to be a kind place, then maybe we ought to have an “unkind” flag.

                                                                                                1. 4

                                                                                                  An “unkind” flag seems like a good idea, and makes more sense to me than “unkind” as a reason for downvoting.

                                                                                              2. 1

                                                                                                “Big meanie”

                                                                                              1. 1

                                                                                                Would you link some example comments you find difficult to classify using the current categories?

                                                                                                  1. 7

                                                                                                    The first link you posted being an example of something that you think should be downvoted is exactly why I’m against this proposal. Maybe it’s rude, but it’s a perfectly valid point if you don’t buy into the “make up your own pronouns and then everyone else has to to humor them or face social punishment” thing. It’s similarly rude to criticize religious dogma, but it’s important that people feel free to do it.

                                                                                                    1. 5

                                                                                                      Regardless of how valid or invalid it is, it’s still a very rude and obnoxious post. “I don’t think people should decide their own pronouns” is something I strongly disagree with but wouldn’t downvote for that. But “schmer ideological agenda from the fact that now paxer is complaining about supposed ‘injustice’ that occurred during xomer time at github” is dragging the discussion down to playground bullying.

                                                                                                      1. 4

                                                                                                        Can you see how <moral system> setting the terms for how you’re allowed to criticize <moral system> is inherently contradictory to productive debate?

                                                                                                        Religions usually forbid making fun under a variety of moral proscriptions like blasphemy, rudeness, or disrespect. However, making fun is one of the best ways to criticize religion. The same is true for the idea that there are 57 genders (or whatever number/cardinality you happen to believe) and that people should have to remember which one every person they know identifies with. It is natural that a stable belief system finds excuses to cut off opposition. In this case, the excuse is that effective opposition is mean/rude/bullying. While I’m sure you earnestly believe that, hopefully you can see that externally to your belief system, this is just the same old defensive censorship we’ve seen for time eternal.

                                                                                                        1. 5

                                                                                                          Okay, I think I’m approaching this as a question of degree, while you seem to be arguing an existential, that “rudeness should be discouraged” is categorically a threat to productive discussion. If that’s not your stance, we’re arguing logistics. If it is, I’d ask you to compare the following.

                                                                                                          “There’s no concrete studies showing static typing meaningfully reduces bugs. As for your examples, there are several reasons they don’t work:…”

                                                                                                          vs

                                                                                                          “That’s the stupidest fucking shit I’ve ever had to read. There’s no evidence static typing does anything for bug reduction and your examples have more mistakes than they have words. Honestly the fact you’re defending it tells me just how bad a ‘programmer’ you are. Why are you even here?”

                                                                                                          Should we, as a community, discourage the latter? Why our why not?

                                                                                                          1. 5

                                                                                                            The latter conveys very little information despite using many words. I would discourage it on the grounds that inefficient communication wastes my time. On the other hand, “you’re wrong. Studies show static typing sucks.” is no doubt rude by some measure, but at least it’s brief. I’m fine with it, though it’s exactly the kind of remark that draws complaints about meanness.

                                                                                                            1. 7

                                                                                                              Arguably that isn’t good because it doesn’t specify why static typing “sucks”, but that’s more a quibble. Personally, I’m fine with “you’re wrong, here’s why”. I’m not fine with “you’re wrong, here’s why, fuck you, fuck off. “

                                                                                                              Communicating is hard and there’s a lot of grey area.

                                                                                                            2. 3

                                                                                                              The latter, in my opinion, clearly falls under “troll”, whereas (for example) using the wrong pronoun would not. I think that is a preferable state of affairs to having “not nice” be its own much broader category.

                                                                                                              1. 3

                                                                                                                My issue with that categorising is that often, the latter is not trolling; full of references, deconstructions, counterpoints, etc. It’s just slathered in insults and venom.

                                                                                                                Look, I’m not saying “everyone’s polite” is the perfect community and there’s absolutely nothing wrong with it. You do lose something by enforcing civility. But the rudeness is okay niche is already filled by HN, proggit, and Facebook, and I think keeping lobsters “nice” is more in line with it’s unique values than not.

                                                                                                                1. 3

                                                                                                                  If it’s designed to provoke a flamey response then it’s trolling, even if it’s true, well-referenced and so on.

                                                                                                    2. 3

                                                                                                      The one that made me consider this was https://lobste.rs/s/js3pbv/antisocial_coding_my_year_at_github#c_fjbnyx, but I’m sure there are others. I don’t think it’s a widespread problem, but it does seem like there’s a hole in the current list of categories.

                                                                                                      1. 5

                                                                                                        What is wrong with that comment, besides the fact that you disagree with it politically?

                                                                                                        1. 2

                                                                                                          If you had to pick from the existing downvote categories, which one-or-two categories best describes your reason for wanting to downvote that comment?

                                                                                                          1. 2

                                                                                                            I would choose “Troll”, though I don’t believe the user is trolling in this particular instance. It’s not exactly off-topic, because it is related to the issue brought up by the original post. It’s not “incorrect” in any technical sense. It’s also clearly not a “me too” post or spam.

                                                                                                            1. 4

                                                                                                              Given that you’re inclined to downvote the comment but can’t find a suitable category to justify that act, do you think you could write a reply articulating your disagreement instead?

                                                                                                              1. 3

                                                                                                                I personally believe that the original post and its associated discussion would be better off without that comment there, rather than another follow-up comment, and the fact that I can’t find an appropriate category is the reason I posted this meta in the first place. I believe the comment should be downvoted because it is unproductive and overly hostile, and it is unclear that a follow-up comment would be any more constructive to the overall discussion of the post. Since the comment is subjective and not based in fact, it is likely that any follow-up would also simply be a subjective response, and that probably wouldn’t help anyone.

                                                                                                                1. 17

                                                                                                                  For what it’s worth, I think that @nickpsecurity there maintained a polite if heated tone–I don’t agree completely with their viewpoint, but that doesn’t matter since they didn’t attack any of the users here directly in the comments.

                                                                                                                  They argued earnestly and in good-faith (see the subthread with @meskarune) and ceded points in a place or two.

                                                                                                                  You seem to be arguing for a downvote category for “I don’t agree with this person”, which is not something that we want or need here.

                                                                                                                  1. 3

                                                                                                                    I agree with you. I would also hate to see this becoming a cencorship-type ordeal. This is also why I only linked to that one comment, as some of the others made in the same thread seemed mostly fine as far as tone is concerned. I object specifically to the linked comment, in particular because it seemed purely inflammatory, and did not contribute positively to the conversation, and I think that’s what such a new downvote category should be limited to. As mentioned elsewhere, I have very rarely seen comments worthy of such a categorization on lobste.rs, but that doesn’t mean it will never happen, nor that the category shouldn’t exist.

                                                                                                                  2. 10

                                                                                                                    I believe the comment should be downvoted because it is unproductive and overly hostile

                                                                                                                    Since the comment is subjective

                                                                                                                    The comment does indeed express a strong subjective opinion, at the very least. I do not see what is wrong with that. Fortunately subjective opinions are (nearly) an inherent element of discourse. And not all statements like “I think this.” need to be productive. Exchange of opinion is just that, exchange of opinion.

                                                                                                                    Your perception that the comment is hostile is also subjective. I prefer that downvotes are reserved for objectively undesirable material. So if someone holds a subjective opinion you do not like, you can ignore that comment and move on. There is no need to attack it with comments or downvotes.

                                                                                                                    1. 5

                                                                                                                      If I understand what you’re after here, you’d like something that in this instance fits between troll and off-topic, but aren’t comfortable with any label you’ve come up with to describe the difference. Given the categories you’ve got today you would categorize the comment troll and replies to it (including yours) off-topic.

                                                                                                                      The term troll has a long history of use online, I’d encourage you to become more comfortable with the label as a description of acting in bad faith. The nature of abuse online, particularly in technical communities, is that strictly operational (objective) definitions of the behavior will lead to bickering over irrelevant details. It can all be described as trolling and the issue then put aside.

                                                                                                                      1. 15

                                                                                                                        a description of acting in bad faith

                                                                                                                        I think a people get hung up on this idea that their motivation matters with the word “troll”. It’s come up every time someone gets called on trash behavior on the site or in the chat, they always say “nuh-uh, I’m not trying to be a jerk” (with varying levels of believability). I’m not particularly interested whether someone is deliberately trolling or just really so clueless/incendiary. I don’t need a Turing test for what’s in a troll’s heart.

                                                                                                                        1. 3

                                                                                                                          That seems reasonable.

                                                                                                                          1. 2

                                                                                                                            is that strictly operational (objective) definitions of the behavior will lead to bickering over irrelevant details. It can all be described as trolling and the issue then put aside.

                                                                                                                            That’s a good point. There’s scores of people that do that on purpose on any moderated forum, esp Facebook, They constantly contrive stuff in any grey areas specifically to incite readers or moderators. Articles on moderation say it’s a constant time-sink on big sites.

                                                                                                                        2. 2

                                                                                                                          I for one do not know how to reply! And I am acutely aware that the thread is part of a culture war, or something… A successful rebuttal would end up on some MRA’s whiteboard, where a team of pros would dissect it and develop an effective counter-rebuttal.

                                                                                                                          *sigh*

                                                                                                                          1. 6

                                                                                                                            Once a conversation has devolved in to talking points between opposing teams the conversation has wandered firmly in to off-topic territory and should be taken to another website that caters to that audience.

                                                                                                                            1. 3

                                                                                                                              Agreed. And are downvotes the appropriate mechanism to encourage that? (Note: I don’t know if the thread in question got all the way there. I’ve buried that tab for later; it is long.)

                                                                                                                              1. 7

                                                                                                                                Downvotes are an appropriate mechanism for comments that “aren’t even wrong.” If I were going to do more than downvote comments as first troll and then off-topic (should a belligerent user persist) it would be to clarify which part of the article they’re replying to. Folk should comment on the material presented, not parrot a talking point that could be trotted out for any article discussing a controversial topic.

                                                                                                                                If they can’t stay on-point, speak to the material presented, and help all of us make sense of what we’re reading then demonstrate that ability for them or upvote other users that are.

                                                                                                                                1. 3

                                                                                                                                  I personally just use the hide button to mute the entire story & all of its comments when that kind of thing starts happening, because I don’t want to see the comments continue to show up in my view of /comments.

                                                                                                                                  1. 3

                                                                                                                                    Well, whether it got too bad or not, I’m still glad you dropped your Trump supporter comment as it made me broaden my use of the term. It was also funny to read when they were summarized so well with same style I labeled SJW’s. The counterpoints in the thread did get me thinking. It’s one of the reasons I participate in them.

                                                                                                                                    That said, I already told jcs and others I’d prefer political stuff to not be on Lobsters at all. Just avoids this stuff entirely since there’s only so many ways a political flamewar can come out of a discussion on algorithmic generation of game content as an example. People would have to work at it to point where they’d be undeniably obvious and easy to filter.

                                                                                                                              2. 1

                                                                                                                                A successful rebuttal would end up on some MRA’s whiteboard, where a team of pros would dissect it and develop an effective counter-rebuttal.

                                                                                                                                The irony of that statement is you’re doing it to me w/ sizeable group of people. I was pretty sure it would happen, too, when I wrote the first comment w/ same worry you express. Odds increased the more I wrote. The good news for you is that, if you supported downvoting that stuff, the a lot of this thread agrees with your position. You don’t have to worry about anything. Only dissenters to a popular belief have to worry. You can relax. :)

                                                                                                                                Note: I’m relatively relaxed, too, given people are being reasonable. It’s been a discussion of community norms more than an attack. Impressively civil and fair, too.

                                                                                                                                EDIT: It just popped into my head this might be seen as an attack or something. Think of it more like me doing an eye roll and then adding perspective to your comment. Maybe worth thinking on what your doing vs what your worried about people doing to you when considering whatever might come out of such situations.