1. 7

    This is great news! Now I can finally recommend OpenBSD to my friends that need to run Apache / PHP / NGINX / Samba :)

    1. 6

      One statistic I’m interested in is the number of submitted stories authored by ppl from the community. I think this is a strong differentiator of Lobste.rs compared to other link aggregation sites.

      1. 3

        There appears to be an increase in them recently, too.

        1. 3

          I had a spot of insomnia and got curious, so I wrote the query. (Reminder I’m happy to run queries, see /about.) The user_is_author boolean was added to stories on 2019-07-30.

          > select extract(year from created_at) as y, extract(month from created_at) as m, sum(user_is_author) as authored, count(*) as "all", round(sum(user_is_author)/count(*)*100) as p
           from stories where created_at >= '2015-07-01' group by 1, 2;
          +------+------+----------+------+------+
          | y    | m    | authored | all  | p    |
          +------+------+----------+------+------+
          | 2015 |    7 |        2 |  608 |    0 |
          | 2015 |    8 |       88 |  541 |   16 |
          | 2015 |    9 |      106 |  549 |   19 |
          | 2015 |   10 |      117 |  737 |   16 |
          | 2015 |   11 |      109 |  791 |   14 |
          | 2015 |   12 |      124 |  761 |   16 |
          | 2016 |    1 |      116 |  972 |   12 |
          | 2016 |    2 |      117 |  849 |   14 |
          | 2016 |    3 |      131 |  736 |   18 |
          | 2016 |    4 |      131 |  739 |   18 |
          | 2016 |    5 |      132 |  786 |   17 |         
          | 2016 |    6 |      157 |  798 |   20 |         
          | 2016 |    7 |      129 |  812 |   16 |                                                          
          | 2016 |    8 |      124 |  797 |   16 |
          | 2016 |    9 |      110 |  731 |   15 |                                                          
          | 2016 |   10 |      115 |  779 |   15 |         
          | 2016 |   11 |      116 |  835 |   14 |  
          | 2016 |   12 |      119 |  852 |   14 |                                                          
          | 2017 |    1 |      152 | 1037 |   15 |       
          | 2017 |    2 |      139 | 1068 |   13 |
          | 2017 |    3 |      182 | 1194 |   15 |  
          | 2017 |    4 |      147 |  947 |   16 |         
          | 2017 |    5 |      160 |  979 |   16 |
          | 2017 |    6 |      175 |  941 |   19 |                                                          
          | 2017 |    7 |      206 | 1109 |   19 |
          | 2017 |    8 |      170 | 1111 |   15 |
          | 2017 |    9 |      202 |  974 |   21 |                                                                                                                                                            
          | 2017 |   10 |      223 |  985 |   23 |                                                          
          | 2017 |   11 |      180 |  924 |   19 |                                                          
          | 2017 |   12 |      165 |  922 |   18 |         
          | 2018 |    1 |      227 |  961 |   24 |                                                          
          | 2018 |    2 |      181 |  846 |   21 |
          | 2018 |    3 |      212 | 1058 |   20 |                                                          
          | 2018 |    4 |      213 |  983 |   22 |         
          | 2018 |    5 |      200 |  982 |   20 |                                                          
          | 2018 |    6 |      188 |  886 |   21 |         
          | 2018 |    7 |      198 | 1017 |   19 |                                                          
          | 2018 |    8 |      207 |  985 |   21 |         
          | 2018 |    9 |      212 |  869 |   24 |                                                                                                                                                            
          | 2018 |   10 |      207 |  918 |   23 |                                                          
          | 2018 |   11 |      221 |  924 |   24 |
          | 2018 |   12 |      225 |  905 |   25 |
          | 2019 |    1 |      218 | 1058 |   21 |
          | 2019 |    2 |      207 |  924 |   22 |
          | 2019 |    3 |      246 |  978 |   25 |
          | 2019 |    4 |      271 |  967 |   28 |
          | 2019 |    5 |      229 |  954 |   24 |
          | 2019 |    6 |      260 |  925 |   28 |
          | 2019 |    7 |       62 |  187 |   33 |
          +------+------+----------+------+------+
          49 rows in set (0.97 sec)
          

          We’ve had rumblings in #lobsters that there’s been too much self-promotion/content marketing recently, and there’s certainly more author-submitted posts the last four months (as @nickpsecurity observed). Maybe someone would like to write queries to get at this?

          1. 3

            Appreciate the data. It confirms that authored by’s went up with it being steadily over 200 since 08/2018. Our curators are pretty good at flagging away, countering or just dropping self promotion with no content. I’m not too worried about it. I probably need to get back to messaging these people about what kinds of content Lobsters prefer. I used to do it sometimes for people whose submissions got no reaction or a negative one.

        1. 3

          This had the useful result that you could no longer accidentally cat a directory and get all sorts of gibberish spewed on your screen, without requiring cat (and everything else that reads files) to explicitly refuse to touch directories. This feature does not seem to have spread to Solaris or the *BSDs, at least as far as I can see.

          Failing a read(2) with EISDIR was introduced in OpenBSD 6.1 (released in 2017). See this comment and the rest of the thread for some additional insights into history.

          1. 2

            As I’m a bit ignorant about these things, just wondering if anyone can explain the motivation for the Xorg setuid change?

            1. 10

              A local root hole in X.org discovered around the release of 6.4: https://marc.info/?l=openbsd-tech&m=154050351216908&w=2

              1. 2

                https://marc.info/?l=openbsd-tech&m=154050351216908&w=2

                Wow, that’s… Very interesting.

                OTOH, I think it’s pretty cool that Theo would reveal such info even though it may seem to undermine credence about revealing security vulnerabilities to his own project. I mean, if his own people don’t tell him about the upcoming embargoed bugs due to the known stance of The OpenBSD Project against security embargoes, what can be expected of other projects and bigger vendors?

                I’m happy to see matthieu is still a committer, though, and has been committing throughout Oct and Nov last year around this controversy, too; i.e., at least from the public eye, there’s no evidence to suggest that his account was ever disabled; but this is some harsh reality come OpenBSD way…

              2. 6

                Heh, I’ve used startx(1) for as long as I’ve used Unix systems, guess I have to finally start using a login manager!

              1. 11

                Some personal favorites:

                • Work has started on a ISC-licensed rsync-compatible program called OpenRSYNC. In this release it has basic functionality such as -a, –delete, but lacks –exclude. Work will continue.
                • unveil(2) has been improved to understand and find covering unveil matches above the working directory of the running process for relative path accesses. As a result many programs now can use unveil in broad ways such as unveil(”/”, “r”).
                • Now using unveil(2) in ospfd(8), ospf6d(8), rebound(8), getconf(1), kvm_mkdb(8), bdftopcf(1), Xserver(1), passwd(1), spamlogd(8), spamd(8), sensorsd(8), snmpd(8), htpasswd(1), ifstated(8). Some pledge(2) changes were required to accommodate unveil.
                • RETGUARD replaces the stack protector on amd64 and arm64, since RETGUARD instruments every function that returns and provides better security properties than the traditional stack protector.
                • tcpdump(8) already used privsep, pledge(2) and unveil(2) containment. It now also drops root privileges completely (switching to a reserved uid).
                • malloc(3) now uses sysctl(2) to get its settings, making it respect the system-wide settings in chroots as well.
                • LibreSSL 2.9.1: Added support for XChaCha20 and XChaCha20-Poly1305.
                • Removed some ASN.1 related code from libcrypto that had not been used since around 2000.
                • ssh(1): When prompting whether to record a new host key, accept the key fingerprint as a synonym for “yes”. This allows the user to paste a fingerprint obtained out of band at the prompt and have the client do the comparison for you.
                • Xorg(1), the X window server, is no longer installed setuid. xenodm(1) should be used to start X.
                • Mandoc 1.14.5: Much better HTML output, in particular with respect to paragraphs, line breaks, and vertical spacing in tagged lists. Tooltips are now implemented in pure CSS, the title attribute is no longer abused.

                I guess the new filtering engine of OpenSMTPD isn’t ready yet.

                1. 14

                  “O Deep Thought computer,” he said, “the task we have designed you to perform is this. We want you to tell us….” he paused, “The Answer.”
                  “The Answer?” said Deep Thought. “The Answer to what?”
                  “Life!” urged Fook.
                  “The Universe!” said Lunkwill.
                  “Everything!” they said in chorus.
                  Deep Thought paused for a moment’s reflection.
                  “Tricky,” he said finally.
                  “But can you do it?”
                  Again, a significant pause.
                  “Yes,” said Deep Thought, “I can do it.”
                  “There is an answer?” said Fook with breathless excitement.
                  “Yes,” said Deep Thought. “Life, the Universe, and Everything. There is an answer. But, I’ll have to think about it.”

                  Fook glanced impatiently at his watch.
                  “How long?” he said.
                  “Seven and a half million years,” said Deep Thought.
                  Lunkwill and Fook blinked at each other.
                  “Seven and a half million years…!” they cried in chorus.
                  “Yes,” declaimed Deep Thought, “I said I’d have to think about it, didn’t I?”

                  [Seven and a half million years later…. Fook and Lunkwill are long gone, but their descendents continue what they started]

                  “We are the ones who will hear,” said Phouchg, “the answer to the great question of Life….!”
                  “The Universe…!” said Loonquawl.
                  “And Everything…!”
                  “Shhh,” said Loonquawl with a slight gesture. “I think Deep Thought is preparing to speak!”
                  There was a moment’s expectant pause while panels slowly came to life on the front of the console. Lights flashed on and off experimentally and settled down into a businesslike pattern. A soft low hum came from the communication channel.

                  “Good Morning,” said Deep Thought at last.
                  “Er..good morning, O Deep Thought” said Loonquawl nervously, “do you have…er, that is…”
                  “An Answer for you?” interrupted Deep Thought majestically. “Yes, I have.”
                  The two men shivered with expectancy. Their waiting had not been in vain.
                  “There really is one?” breathed Phouchg.
                  “There really is one,” confirmed Deep Thought.
                  “To Everything? To the great Question of Life, the Universe and everything?”
                  “Yes.”
                  Both of the men had been trained for this moment, their lives had been a preparation for it, they had been selected at birth as those who would witness the answer, but even so they found themselves gasping and squirming like excited children.
                  “And you’re ready to give it to us?” urged Loonsuawl.
                  “I am.”
                  “Now?”
                  “Now,” said Deep Thought.
                  They both licked their dry lips.
                  “Though I don’t think,” added Deep Thought. “that you’re going to like it.”
                  “Doesn’t matter!” said Phouchg. “We must know it! Now!”
                  “Now?” inquired Deep Thought.
                  “Yes! Now…”
                  “All right,” said the computer, and settled into silence again. The two men fidgeted. The tension was unbearable.
                  “You’re really not going to like it,” observed Deep Thought.
                  “Tell us!”
                  “All right,” said Deep Thought. “The Answer to the Great Question…”
                  “Yes..!”
                  “Of Life, the Universe and Everything…” said Deep Thought.
                  “Yes…!”
                  “Is…” said Deep Thought, and paused.
                  “Yes…!”
                  “Is…”
                  “Yes…!!!…?”
                  “the new filtering engine of OpenSMTPD isn’t ready yet,” said Deep Thought, with infinite majesty and calm.

                  1. 2

                    Is this the new “Is KERNSEAL ready yet?”

                1. 18

                  I have long had trouble understanding how the blockchain gets the very many magical properties ascribed to it. Every time I’ve asked, I have gotten a lecture in hashes and proof of work. I know the fundamentals of blockchains, what I don’t understand is how the fundamentals lead to these amazing emergent properties.

                  This article sorta kinda makes me think I might not be missing anything at all – the people talking about it may have been full of shit.

                  1. 8

                    I have been wrong before but to me it seems like a mass psychological phenomenon. That many people and that much money cannot be wrong! So they add more people and money.

                    In the best case, some companies use the label “block chain” to market some established cryptographic techniques that are not block chain at all…

                    1. 7

                      yep. You might think “blockchain” meant something like “append-only ledger with a consensus mechanism”, but it turns out in practice to literally just mean “whatever I’m trying to sell you today”.

                      I was talking about this a few months ago with a well-meaning non-techie, who suggested that Uber - the money-burning minicab firm with an app - was an example of a “decentralised system.” More than that - it was a model for how blockchain could succeed.

                      I think they’d never thought about the concept of distributed systems of any sort ever before in their lives.

                      “It’s like blockchain, because anyone can sign up to be an Uber driver!”
                      “Uh … anyone can sign up to be a minicab driver.”

                      or the very concept of “open source” only being possible with “blockchain”.

                      1. 4

                        The weird thing is that I know intelligent, technical people that advocate for this. If asked for specifics, some variant of “we still have to figure out the specifics” is used.

                        Well, chances are that you never will…

                      2. 4

                        The hype cycle became self fulfilling. I got a look at the internal roadmap for one of the pieces of legacy software at the big enterprise I work at - crusty old barely touched 90s technology that’s critical for parts management and ordering.

                        2020 plans? Traceability of parts on the blockchain.

                        1. 1

                          Traceability, correct me if I’m wrong, was one of the actual things a distributed append-only ledger was good at. The way I see it, it’s a good decision with regards to what tech to use, at least until someone puts the wrong data in.

                      3. 4

                        As well as that non-tech explanation talk, I have the longer and more techy version to an infosec group. (My mission to get across to them: “please don’t get into blockchains”)

                        1. 4

                          You’re sending the wrong message. Instead, tell them to come up with something useful, pitch a blockchain version, build the useful thing first with the money, open source it, and then build the “better” version with blockchain. We’ll steadily get more useful stuff out of blockchain investments.

                        2. 4

                          I’d like to refer to this article about Bitcoin from 2011, before all the mass hysteria began: https://paulbohm.com/articles/bitcoins-value-is-decentralization/

                          To elaborate: Bitcoin isn’t just a currency but an elegant universal solution to the Byzantine Generals’ Problem, one of the core problems of reaching consensus in Distributed Systems. Until recently it was thought to not be practically solvable at all, much less on a global scale. Irrespective of its currency aspects, many experts believe Bitcoin is brilliant in that it technically made possible what was previously thought impossible.

                          /edit quote

                          1. 4

                            Herd behavior. It’s usually irrational except for the scheming people fueling and benefiting from it.

                            Blockchain looks like herd behavior. Similarly, most of it has a tiny group of people that will get rich if enough buy in. That’s similar to how companies like Goldman create, pop, and profit from bubbles in the market.

                            1. 3

                              irrational exuberance meets unjustifi-ed/able faith in technology.

                              if you bought into a blockchain, you want to hype it up because that’s how you get paid. If you didn’t buy into it, well you got bored of trying to reason with people a long time ago.

                              It’s probably the most interesting social phenomenon of recent years.

                              1. 2

                                It sounds basically like a pyramid scheme when you put it like that…

                              2. 2

                                The flip side of this is that some companies are actually trying to look for the blockchain-based “killer app”, if such an app ever exists. I did develop a few blockchain based proof of concepts, which didn’t go anywhere, but there wasn’t any attempt to trick anyone. It’s just about experimenting with a new technology and see what can be done with it.

                              1. 1

                                I left a Git branch stall for a few weeks and, when I got back to it, it was behind by almost 60,000 commits. I think one can argue that a single person cannot possibly read all the code that is added to Windows every day, let alone read what was written during the past thirty years!

                                What about peer review and quality assurance?!

                                1. 28

                                  It’s entirely possible that Microsoft has more than one person doing peer review and QA on Windows.

                                  1. 3

                                    Given that it is a monorepo, the vast majority of those commits are probably not touching the area of the codebase that you’re personally working on.

                                  1. 33

                                    Am I missing something, or is that a whole article? Looks like an excerpt or preface though :)

                                    But from what I see, your problem isn’t a vim itself, but rather plugins and overloading yourself with them. A common mistake for Vim newcomers is that they come in and just skip the vi grammar part (calling it “the boring theory stuff”) and trying things too hard. Then they discover plugins and – more importantly – plugin managers, which lower the bar dramatically. Within such conditions, they start to feel “safe” and install as many plugins as they can handle to mimic their previous IDE, their colleagues’ IDE or any IDE they pretend to be better from, just to show off maybe.

                                    And after all, you end up with a pretty much beefied up editor which you don’t really know. Okay, well, you know how to open it, type the text, save and call it a day. You know “how”, but now “why”.

                                    In every editor which has been thought reasonably from the ground up (not just vi - it could be emacs, joe, sam, acme, kakoune, vis, CygnusED, EDLIN…) you need to understand the rules. It’s like speaking in new dialect of your language or swimming - won’t come overnight which sounds unpleasant in age of instant grafitication, but its long-term benefits are hard to overlook.

                                    So, in other words (or as people say these days - tl;dr), your problem is that you don’t grok vi (or any other editor, as I said).

                                    Of course, if you want to just click ok and go on with things, you might just spawn up your web browser^W^Wtext editor called vscode, throw a ton of plugins on yourself and cheer. Would be that faster? Maybe. But only in short-term goal or if your project isn’t going to last more than year or two (which is a standard nowadays). Or if you just don’t want to “spend time on these neckbeardy things”.

                                    1. 3

                                      So, in other words (or as people say these days - tl;dr), your problem is that you don’t grok vi (or any other editor, as I said).

                                      I’m not sure how these words hold up to someone who calls himself a “VIM master from using others’ plugins to rolling out my own”.

                                      After seeing some colleagues working at lightning speed with PhpStorm I gave up thinking vi(m) is always faster, especially on the cases the OP mentions like renaming/refactoring classes, functions and files in a big project. The reason I still use vi for everything, is because today I mainly develop in C, which is natively supported in vi and because I can use vi for everything, not only development but also sysadmin, mail composing etc.

                                      1. 3

                                        You said what I was thinking in more words.

                                        1. 2

                                          Am I missing something, or is that a whole article? Looks like an excerpt or preface though :)

                                          This is ‘part 1’, so I presume they split it up to increase their exposure, which means we will probably see part 2, 3, 4,…,N of this poor article here over time.

                                        1. 5

                                          However, benchmarks quickly showed that wireguard-go falls very short of the performance offered by the kernel module. This is because while the Go language is very good for writing servers, it is not so good for raw packet processing, which a VPN essentially does.

                                          Is the difference between Go versus Rust really that big? I would suspect the difference between kernel and userland is way bigger.

                                          Rust is a modern, safe language that is both as fast as C++ and is arguably safer than Go (it is memory safe and also imposes rules that allow for safer concurrency), […].

                                          Interesting point about Rust being arguably safer than Go.

                                          1. 12

                                            Go is pretty fast and more than enough for lots of tasks, especially in its niche of higher-level network-bound servers.

                                            However, in this case there’s a bunch of cryptography, packet parsing and shuffling of data from one place to another, where LLVM optimized code, zero-cost abstractions and no-GC memory slices shine.

                                          1. 3

                                            Though there are already objections to do ECDH and key derivation in the kernel

                                            I’ve read some objections by @zx2c4 as well about putting timers and handshake code outside of the kernel when this was proposed by someone developing a kernel version for OpenBSD: https://lists.zx2c4.com/pipermail/wireguard/2018-December/003656.html

                                            In my own userland implementation I’ve separated the long term secrets into a small privileged process apart from the rest.

                                            1. 2

                                              In my own userland implementation I’ve separated the long term secrets into a small privileged process apart from the rest.

                                              That’s the right way to do it. :)

                                              EDIT: Finally reading mailing list (shorter than I thought), his counter is pretty good. We keep stuff out of the kernel by default due to higher damage upon code injection. Another principle for trusted code is keeping it simple enough to know with confidence that won’t happen. He says he did that. The last factor is whether it’s easier to analyze his code for immunity vs yours plus its interactions with kernel code. The latter might be considerably more complex. This is good example of where exception might be made. So, either sounds good in this case. One might still apply some verification or hard-hitting analyzers to it if wanting extra assurance.

                                            1. 4

                                              So I wasn’t sure whether or not to say I’m the author here. I’m not the author, but I am running the instance of writefreely on OpenBSD. I’m hoping to build this into a federated space for others interested in reading and writing medium-high quality content, particularly around the BSD community.

                                              I’ll have an open beta from April, but if anyone’s interested in writing content here, drop me a line for an invite. It’s totally open, all your data is yours, you can read content without javascript and there’s no visitor tracking beyond post view counts accessible only by a post’s individual author.

                                              It’d be really nice to get some people interested in writing about lesser known operating system experiences, Open/NetBSD use, less common FreeBSD use and so on.

                                              1. 2

                                                “So I wasn’t sure whether or not to say I’m the author here. I’m not the author”

                                                You’re the author if you wrote the content, built the tools, etc. That’s probably the original intent. It’s also the usage in almost all situations I see “authored by.”

                                                1. 3

                                                  Yeah it was h3artbl33d that wrote the piece, hit publish, I just gave him an account. It’s the submit tick box I’m unsure about. I don’t want to claim authorship, but certainly there’s a connection and I’m not sure the best way to make that clear.

                                                  1. 1

                                                    Maybe you/chargen.one can be seen as a publisher, comparable to the publisher of a journal.

                                              1. 39

                                                I’m really burning out on “simplicity” posts. I get it, simplicity is good. But that doesn’t actually inform me as a developer. Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                                It’s like if I stood on stage and said “Be good! Don’t be evil! Being evil is bad!” Sure, everybody agrees with that, but does it actually help people make moral choices?

                                                (Also the analogy is dumb. Yes, we should totally base our engineering practice on a movie! A movie where the engineers are wrong because of magic.)

                                                1. 11

                                                  Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                                  Because your questions are difficult and answers are dependent on a lot of factors.

                                                  I’ll tell you what I do to detect simplicity, maybe you’ll find it useful. Let’s start with a real-life example.

                                                  I needed tokens for authorization, I reviewed existing formats, JWTs look conservative and Macaroons look powerful.

                                                  What do I do? I dissect the formats. For JWTs I read the RFCs and implemented software to create them and verify them (each in 2 languages) for various options (key algorithms).

                                                  For Macaroons I read the whitepaper, then implemented verifier based on the whitepaper, reviewed existing implementations, found out differences between the whitepaper and de-facto code with explanations. While comparing my implementation I found out some security issues with existing code. Additionally I implemented the rest of the stack (de/serialization, UI for manipulation of Macaroons). After two months I knew precisely where does complexity lie in Macaroons and of course there are the only spots all blogposts don’t mention (spoilers: cycles in third party caveats, no standards for encoded caveats…)!

                                                  Then I looked at my JWT proof-of-concept code - it uses base64(url) and JSON, primitives that basically all programming environments have built-in. After limiting the algorithms used the entire verifier takes just a couple of lines of code! It’s vastly simpler than the Macaroon one.

                                                  What’s the moral here? That you need a lot of time to see for yourself what is simple and what is complex. Now every time I see a post recommending Macaroons I can already see the author didn’t use them in practice (compare that with the Tess Rinearson post linked at the end of that article).

                                                  That’s only the example, I routinely implement various protocols and re-implement software (ActivityPub, Mailing Lists, roughtime client) and each time I discover what’s simple or what’s complex in each one of them.

                                                  (By the way your book is excellent!)

                                                  1. 9

                                                    I get it, simplicity is good.

                                                    Alas, not everybody gets it. The best that these kinds of exhortations can do (all that they aim to do, as far as I can tell) is to persuade people to modify their own set of values. This doesn’t immediately result in better code… but I think it’s a necessary precondition. The only developers who will even ask the good questions you suggest (let alone look for good answers) are the developers who hold simplicity as a value.

                                                    (The analogy is pretty dumb though, and not especially motivating.)

                                                    1. 10

                                                      I’ve never met a developer who does not claim to hold simplicity as a value. But as a concept it is so subjective that this is meaningless. It’s extremely common for two developers arguing for opposing approaches each to claim that their approach is the simpler one.

                                                      1. 7

                                                        I get the value of exhortations. I think more examples would be better. Pairs of solutions where the simple one meets requirements with a number of better attributes. Developers often prefer to see the difference and benefits instead of being told.

                                                      2. 6

                                                        Exactly. This is one of those things you can’t explain in a book. When to compose, when to decompose. When to extract methods, when to inline methods. When to add a layer of abstraction, when to remove one. When is it too flexible, when is it too simplistic?

                                                        No amount of rules of thumb is going to answer those question. I only know of one way to learn it: practice. Which takes effort and most importantly, time. Rendering this kind of posts mostly useless.

                                                        1. 3

                                                          P.S. They do feel good to write though, so people will keep writing them, and there’s nothing wrong with it either.

                                                        2. 5

                                                          I agree that anecdotes like this can get old, but I’ve been meaning to actually write a similar post to this… on something I’ve been calling the “too many buttons” syndrome. This issue pops up a ton in large pieces of software (Though I’m specifically thinking of projects like JRA and Confluence) where there’s an option for everything.

                                                          Not everyone gets that simplicity is good because it can be harder to sell. “If a user wants it, we should do it” is something I’ve heard just a few too many times without bothering to look at the use case or if it could be done better. Sometimes it’s worth stepping back and looking at the complexity something will add to the project (in both code and testing… especially when it comes to options and how they interact with each other) rather than just adding all the little features.

                                                          1. 5

                                                            In my experience a lot of commercial companies that develop under tight deadlines produce a lot of suboptimal and dreadful code. Often it takes more time, to produce less code simply because the more time you spend on a difficult problem, the better you understand it. I think the reason that most a lot of software is bloated and complex is because it’s “good enough” which is optimal from an economic point of view.

                                                            The other day there was a discussion here on Lobsters about all the required pieces needed to run a Mastodon instance and the popular solution of abstracting all that away in a Docker container. There are alternative implementations that depend on a smaller number of components alleviating the need for dumping everything in a container (of course the question is, do these alternatives offer the same functionality).

                                                            How do we detect simplicity?

                                                            For me personally simplicity has to do with readability, maintainability and elegance of code or infrastructure. If someones solution involves three steps, and someone else can do it in two steps (with comparable cognitive load per step), I would say it’s more simple.

                                                            How do we know when we shouldn’t simplify?

                                                            If that would cut some features you cannot miss.

                                                            1. 5

                                                              You are so right. After years of experience, I only start to clarify my idea of “simplicity”. There are different kind of simplicity most of them are not totally compatible. And in my opinion some need to be preferred to other, but there is no clear rule. To make a choice between different complexity I still use a lot of intuition and I debate a lot, and I am still unsure my choice are the best.

                                                              • only using basic feature of a language (do not use advanced programming language feature) is certainly the most important aspect in simplicity. It will make your code easy to read by more people.
                                                              • don’t use too much intermediate functions, and if possible don’t disperse those function in many different files before really feel you are copy/pasting too much. My rule of thumb is, 2 or 3 times duplications is totally fine and superior to centralisation of code. It start to be really clear that code factorisation is good when you start repeating yourself more than 6 to 10 times
                                                              • only really use advanced feature of the language after having tried not to use it for some time and really lack the ability of that advanced feature. Some examples of what I call advanced feature of a language are; class heritage, protocols in Clojure, writing your own typeclasses in Haskell, meta programming (macros in LISP), etc…
                                                              • prefer stateless functions to objects/service with internal states
                                                              • prefer pure functions (side effect free) other procedures (functions with side effects)
                                                              • give a lot of preference to composable solutions ; composable in the algebraic meaning. For example, I do my best not to use LISP macros, because most of the time macros break composability. The same could be said when you start to deal with type-level programming in Haskell, or when you are doing meta-programming in ruby/python.

                                                              For now, all those rules are still quite artisanal. I don’t have any really hard metrics or strong rules. Everything I just said is “preferable” but I’m pretty sure we can find exception to most of those rules.

                                                              1. 5

                                                                Amen, +1, etc. “Simplicity” often just means that a concept fits cleanly in the maker’s head at a particular point in time. How many times have I returned to a project I thought was simple only to find I had burdened it with spooky magic because I didn’t benefit from critical distance at the time? When was the last time I deemed another person’s work “too complex” because I couldn’t understand it in one sitting and wasn’t aware of the constraints they were operating under? Answers: too often and too recently.

                                                                1. 3

                                                                  What kinds of simplicity are there?

                                                                  This is a good question (as are the others). Borrowing from Holmes, I’d say there’s a continuum from naive simplicity, to complexity, to simplicity on the other side of complexity (which is what is truly interesting)

                                                                  For example, “naively simple” code would only cover a small subset (say, the happy path) of a business problem. Complex code would handle all, or most, of the business complexity but in a messy, complicated way. “Other side” simplicity refines that complex code into something that can handle the business complexity without itself becoming overly complicated.

                                                                  1. 2

                                                                    What happens to simplicity? We trade it for a other things of course. For example, you can have simple regular expressions, but most people prefer less simple and more powerful implementation like Perls.

                                                                    Simplicity is often a tradeoff versus easyness, performance, flexibility, reusability, useability, etc. So simplicity is good, but those other things are also good.

                                                                    1. 1

                                                                      Most people seem to agree that simplicity is best. However, when it comes down to simplicity for the user versus the developer, I have seen disagreement. Each trade off is going to be situation and implementation dependent, but at my job I’ve been pushing for a simpler developer environment.

                                                                      In my office, there is a tendency to create exceptions to rules because it makes things simpler for the user. Since the environment has more exceptional circumstances, it tends to have more errors when people forget the undocumented exception case. In my opinion, this causes an uneven experience for the user despite being “simpler.”

                                                                      My experience is coming from a medium sized, non-tech company. I work in the IT department so we are a cost center. There is an emphasis on white glove treatment of the revenue producing portions of the company. YMMV

                                                                    1. 65

                                                                      In the Mastodon universe, technically-minded users are encouraged to run their own node. Sounds good. To install a Mastodon node, I am instructed to install recent versions of

                                                                      • Ruby
                                                                      • Node.JS
                                                                      • Redis
                                                                      • PostgreSQL
                                                                      • nginx

                                                                      This does not seem like a reasonable set of dependencies to me. In particular, using two interpreted languages, two databases, and a separate web server presumably acting as a frontend, all seems like overkill. I look forward to when the Mastodon devs are able to tame this complexity, and reduce the codebase to a something like single (ideally non-interpreted) language and a single database. Or, even better, a single binary that manages its own data on disk, using e.g. embedded SQLite. Until then, I’ll pass.

                                                                      1. 22

                                                                        Totally agree. I heard Pleroma has less dependencies though it looks like it depends a bit on which OS you’re running.

                                                                        1. 11

                                                                          Compared to Mastodon, Pleroma is a piece of cake to install; I followed their tutorial and had an instance set up and running in about twenty minutes on a fresh server.

                                                                          From memory all I needed install was Nginx, Elixir and Postgres, two of which were already set up and configured for other projects.

                                                                          My server is a quad core ARMv7 with 2GB RAM and averages maybe 0.5 load when I hit heavy usage… it does transit a lot of traffic though, since the 1st January my server has pushed out 530GB of traffic.

                                                                          1. 2

                                                                            doesnt Elixir require Erlang to run?

                                                                            1. 2

                                                                              It does. Some linux distributions will require adding the Erlang repo before installing elixir but most seem to have it already included: https://elixir-lang.org/install.html#unix-and-unix-like meaning its a simple one line command to install e.g pkg install elixir

                                                                          2. 7

                                                                            I’m not a huge social person, but I had only heard of Pleroma without investigating it. After looking a bit more, I don’t really understand why someone would choose Mastodon over Pleroma. They do basically the same thing, but Pleroma takes less resources. Anyone who chose Mastodon over Pleroma have a reason why?

                                                                            1. 6

                                                                              Mastodon has more features right now. That’s about it.

                                                                              1. 4

                                                                                Pleroma didn’t have releases for a looong time. They finally started down that route. They also don’t have official Docker containers and config changes require recompiling (just due to the way they have Elixir and builds setup). It was a pain to write my Docker container for it.

                                                                                Pleroma also lacks moderation tools (you need to add blocked domains to the config), it doesn’t allow remote follow/interactions (if you see a status elsewhere on Mastodon, you can click remote-reply, it will ask your server name, redirect you to your server and then you can reply to someone you don’t follow) and a couple of other features.

                                                                                Misskey is another alternative that looks promising.

                                                                                1. 2

                                                                                  it doesn’t allow remote follow/interactions (if you see a status elsewhere on Mastodon, you can click remote-reply, it will ask your server name, redirect you to your server and then you can reply to someone you don’t follow)

                                                                                  I think that might just be the Pleroma FA - if I’m using the Mastodon FE, I get the same interaction on my Pleroma instance replying to someone on a different instance as when I’m using octodon.social (unless I’m radically misunderstanding your sentence)

                                                                                  1. 1

                                                                                    Thanks, this is a really great response. I actually took a quick look at their docs and saw they didn’t have any FreeBSD guide set up, so I stopped looking. I use Vultr’s $2.50 FreeBSD vps and I didn’t feel like fiddling with anything that particular night. I wish they did have an official docker container for it.

                                                                                  2. 3

                                                                                    Pleroma has a bunch of fiddly issues - it doesn’t do streaming properly (bitlbee-mastodon won’t work), the UI doesn’t have any “compose DM” functionality that I can find, I had huge problems with a long password, etc. But they’re mostly minor annoyances than show stoppers for now.

                                                                                  3. 7

                                                                                    It doesn’t depend - they’ve just gone further to define what to do for each OS!

                                                                                    1. 4

                                                                                      I guess it’s mainly the ImageMagick dependency for OpenBSD that got me thinking otherwise.

                                                                                      OpenBSD

                                                                                      • elixir
                                                                                      • gmake
                                                                                      • ImageMagick
                                                                                      • git
                                                                                      • postgresql-server
                                                                                      • postgresql-contrib

                                                                                      Debian Based Distributions

                                                                                      • postgresql
                                                                                      • postgresql-contrib
                                                                                      • elixir
                                                                                      • erlang-dev
                                                                                      • erlang-tools
                                                                                      • erlang-parsetools
                                                                                      • erlang-xmerl
                                                                                      • git
                                                                                      • build-essential
                                                                                      1. 3

                                                                                        imagemagick is purely optional. The only hard dependencies are postgresql and elixir (and some reverse proxy like nginx)

                                                                                        1. 4

                                                                                          imagemagick is strongly recommended though so you can enable the Mogrify filter on uploads and actually strip exif data

                                                                                    2. 3

                                                                                      Specifically, quoting from their readme:

                                                                                      Pleroma is written in Elixir, high-performance and can run on small devices like a Raspberry Pi.

                                                                                      As to the DB, they seem to use Postgres.

                                                                                      The author of the app posted his list of differences, but I’m not sure if it’s complete and what it really means. I haven’t found a better comparison yet, however.

                                                                                    3. 16

                                                                                      Unfortunately I have to agree. I self-host 99% of my online services, and sysadmin for a living. I tried mastodon for a few months, but its installation and management process was far more complicated than anything I’m used to. (I run everything on OpenBSD, so the docker image isn’t an option for me.)

                                                                                      In addition to getting NodeJS, Ruby, and all the other dependencies installed, I had to write 3 separate rc files to run 3 separate daemons to keep the thing running. Compared to something like Gitea, which just requires running a single Go executable and a Postgres DB, it was a massive amount of toil.

                                                                                      The mastodon culture really wasn’t a fit for me either. Even in technical spaces, there was a huge amount of politics/soapboxing. I realized I hadn’t even logged in for a few weeks so I just canned my instance.

                                                                                      Over the past year I’ve given up on the whole social network thing and stick to Matrix/IRC/XMPP/email. I’ve been much happier as a result and there’s a plethora of quality native clients (many are text-based). I’m especially happy on Matrix now that I’ve discovered weechat-matrix.

                                                                                      I don’t mean to discourage federated projects like Mastodon though - I’m always a fan of anything involving well-known URLs or SRV records!

                                                                                      1. 11

                                                                                        Fortunately the “fediverse” is glued by a standard protocol (ActivityPub) that is quite simple so if one implementation (e.g. Mastodon) doesn’t suit someone’s needs it’s not a big problem - just searching for a better one and it still interconnects with the rest of the world.

                                                                                        (I’ve written a small proof-of-concept ActivityPub clients and servers, it works and federates, see also this).

                                                                                        For me the more important problems are not implementation issues with one server but rather design issues within the protocol. For example established standards such as e-mail or XMPP have a way to delegate responsibility of running a server of a particular protocol but still use bare domain for user identifies. In e-mail that is MX records in XMPP it’s DNS SRV records. ActivityPub doesn’t demand anything like it and even though Mastodon tries to provide something that would fix that issue - WebFinger, other implementations are not interested in that (e.g. Pleroma). And then one is left with instances such as “social.company.com”.

                                                                                        For example - Pleroma’s developer’s id is lain@pleroma.soykaf.com.

                                                                                        1. 16

                                                                                          This is a completely reasonable and uncontroversial set of dependencies for a web app. Some of the largest web apps on the Internet run this stack. That is a good thing, because when Fediverse nodes need to scale there are well-understood ways of doing it.

                                                                                          Success in social networking is entirely about network effects and that means low barrier to entry is table stakes. Yeah, it’d be cool if someone built the type of node you’re talking about, but it would be a curiosity pursued only by the most technical users. If that were the barrier to entry for the network, there would be no network.

                                                                                          1. 39

                                                                                            This is a completely reasonable and uncontroversial set of dependencies for a web app. Some of the largest web apps on the Internet run this stack.

                                                                                            Yes, but not for a web app I’m expected to run on my own time, for fun.

                                                                                            1. 6

                                                                                              I’m not sure that’s the exact expectation, that we all should run our single-user Mastodon instances. I feel like the expectation is that sysadmin with enough knowledge will maintain an instance for many users. This seems to be the norm.

                                                                                              That, or you go to Mastohost and pay someone else for your own single-user instance.

                                                                                              1. 2

                                                                                                You’re not expected to do that is my point.

                                                                                              2. 16

                                                                                                completely reasonable and uncontroversial

                                                                                                Not true. Many people are complaining about the unmanaged proliferation of dependencies and tools. Most projects of this size and complexity don’t need more than one language, bulky javascript frameworks, caching and database services.

                                                                                                This is making difficult to package Mastodon and Pleroma in Debian and Ubuntu and making it more difficult for people to make the service really decentralized.

                                                                                                1. 1

                                                                                                  I’m not going to defend the reality of what NPM packaging looks like right now because it sucks but that’s the ecosystem we’re stuck with for the time being until something better comes along. As with social networks, packaging systems are also about network effects.

                                                                                                  But you can’t deny that this is the norm today. Well, you can, but you would be wrong.

                                                                                                  This is making difficult to package Mastodon and Pleroma in Debian and Ubuntu

                                                                                                  I’m sure it is, because dpkg is a wholly unsuitable tool for this use-case. You shouldn’t even try. Anyone who doesn’t know how to set these things up themselves should use the Docker container.

                                                                                                  1. 1

                                                                                                    I think the most difficult part of the Debian packaging would be the js deps, correct?

                                                                                                    1. 3

                                                                                                      Yes and no. Unvendorizing dependencies is done mostly for security and requires a lot of work depending on the amount of dependencies. Sometimes js libraries don’t create serious security concerns because they are only run client-side and can be left in vendorized form.

                                                                                                      The Ruby libraries can be also difficult to unvendorize because many upstream developers introduce breaking changes often. They care little about backward compatibility, packaging and security.

                                                                                                      Yet server-side code is more security-critical and that becomes a problem. And it’s getting even worse with new languages that strongly encourage static linking and vendorization.

                                                                                                      1. 1

                                                                                                        I can’t believe even Debian adopted the Googlism of “vendor” instead of “bundle”.

                                                                                                        That aside, Rust? In Mastodon? I guess the Ruby gems it requires would be the bigger problem?

                                                                                                        1. 2

                                                                                                          The use of the word is mine: I just heard people using “vendor” often. It’s not “adopted by Debian”.

                                                                                                          I don’t understand the second part: maybe you misread Ruby for Rust in my text?

                                                                                                          1. 1

                                                                                                            No, I really just don’t know what Rust has to do with Mastodon. There’s Rust in there somewhere? I just didn’t notice.

                                                                                                            1. 2

                                                                                                              AFAICT there is no Rust in the repo (at least at the moment).

                                                                                                              1. 1

                                                                                                                Wow, I’m so dumb, I keep seeing Rust where there is none and misunderstanding you, so sorry!

                                                                                                  2. 7

                                                                                                    Great. Then have two implementations, one for users with large footprints, and another for casual users with five friends.

                                                                                                    It is a reasonable stack if you will devote 1+ servers to the task. Not for something you might want to run on your RPI next to your irc server (a single piece of software in those stacks too)

                                                                                                    1. 4

                                                                                                      Having more than one implementation is healthy.

                                                                                                      1. 2

                                                                                                        Of course it is. Which is why it’s a reasonable solution to the large stack required by the current primary implementation.

                                                                                                  3. 6

                                                                                                    There’s really one database and one cache there. I mean, I guess technically Redis is a database, but it’s almost always used for caching and not as a DB layer like PSQL.

                                                                                                    You can always write your own server if you want in whatever language you choose if you feel like Ruby/Node is too much. Or, like that other guy said, you can just use Docker.

                                                                                                    1. 4

                                                                                                      There’s really one database and one cache there. I mean, I guess technically Redis is a database, but it’s almost always used for caching . . .

                                                                                                      A project that can run on a single instance of the application binary absolutely does not need a cache. Nor does it need a pub/sub or messaging system outside of its process space.

                                                                                                      1. 2

                                                                                                        It’s more likely that Redis is being used for pub/sub messaging and job queuing.

                                                                                                      2. 11

                                                                                                        This does not seem like a reasonable set of dependencies to me

                                                                                                        Huh. I must be just used to this, then. At work I need to use or at least somewhat understand,

                                                                                                        • Postgres
                                                                                                        • Python 2
                                                                                                        • Python 3
                                                                                                        • Django
                                                                                                        • Ansible
                                                                                                        • AWS
                                                                                                        • Git (actually, Mercurial, but this is my choice to avoid using git)
                                                                                                        • Redis
                                                                                                        • Concourse
                                                                                                        • Docker
                                                                                                        • Emacs (My choice, but I could pick anything else)
                                                                                                        • Node
                                                                                                        • nginx
                                                                                                        • Flask
                                                                                                        • cron
                                                                                                        • Linux
                                                                                                        • RabbitMQ
                                                                                                        • Celery
                                                                                                        • Vagrant (well, optional, I actually do a little extra work to have everything native and avoid a VM)
                                                                                                        • The occasional bit of C code

                                                                                                        and so on and so forth.

                                                                                                        Do I just work at a terrible place or is this a reasonable amount of things to have to deal with in this business? I honestly don’t know.

                                                                                                        To me Mastodon’s requirements seem like a pretty standard Rails application. I’m not even sure why Redis is considered another db – it seems like an in-memory cache with optional disk persistence is a different thing than a persistent-only RDBMS. Nor do I even see much of a problem with two interpreted languages – the alternative would be to have js everywhere, since you can’t have Python or Ruby in a web browser, and js just isn’t a pleasant language for certain tasks.

                                                                                                        1. 38

                                                                                                          I can work with all that and more if you pay me. For stuff I’m running at home on my own time, fuck no. When I shut my laptop to leave the office, it stays shut until I’m back again in the morning, or I get paged.

                                                                                                          1. 2

                                                                                                            So is Mastodon unusual for a Rails program? I wonder if it’s simply unreasonable to ask people to run their own Rails installation. I honestly don’t know.

                                                                                                            Given the amount of Mastodon instances out there, though, it seems that most people manage. How?

                                                                                                            1. 4

                                                                                                              That looks like a bog-standard, very minimal rails stack with a JS frontend. I’m honestly not sure how one could simplify it below that without dropping the JS on the web frontend and any caching, both of which seem like a bad idea.

                                                                                                              1. 7

                                                                                                                There’s no need to require node. The compilation should happen at release time, and the release download tarball should contain all the JS you need.

                                                                                                                1. -3

                                                                                                                  lol “download tarball”, you’re old, dude.

                                                                                                                  1. 7

                                                                                                                    Just you wait another twenty years, and you too will be screaming at the kids to get off your lawn.

                                                                                                                2. 2

                                                                                                                  You could remove Rails and use something Node-based for the backend. I’m not claiming that’s a good idea (in fact it’s probably not very reasonable), but it’d remove that dependency?

                                                                                                                  1. 1

                                                                                                                    it could just have been a go or rust binary or something along those lines, with an embedded db like bolt or sqlite

                                                                                                                    edit: though the reason i ignore mastodon is the same as cullum, culture doesn’t seem interesting, at least on mastodon.social

                                                                                                                  2. 4

                                                                                                                    If security or privacy focused, I’d try a combo like this:

                                                                                                                    1. Safe language with minimal runtime that compiles to native code and Javascript. Web framework in that language for dynamic stuff.

                                                                                                                    2. Lwan web server for static content.

                                                                                                                    3. SQLite for database.

                                                                                                                    4. Whatever is needed to combine them.

                                                                                                                    Combo will be smaller, faster, more reliable, and more secure.

                                                                                                                    1. 2

                                                                                                                      I don’t think this is unusual for a Rails app. I just don’t want to set up or manage a Rails app in my free time. Other people may want to, but I don’t.

                                                                                                                  3. 7

                                                                                                                    I don’t think it’s reasonable to compare professional requirements and personal requirements.

                                                                                                                    1. 4

                                                                                                                      The thing is, Mastodon is meant to be used on-premise. If you’re building a service you host, knock yourself out! Use 40 programming languages and 40 DBs at the same time. But if you want me to install it, keep it simple :)

                                                                                                                      1. 4

                                                                                                                        Personally, setting up all that seems like too much work for a home server, but maybe I’m just lazy. I had a similar issue when setting up Matrix and ran into an error message that I just didn’t have the heart to debug, given the amount of moving parts which I had to install.

                                                                                                                        1. 3

                                                                                                                          If you can use debian, try installing synapse via their repository, it works really nice for me so far: https://matrix.org/packages/debian/

                                                                                                                          1. 1

                                                                                                                            Reading other comments about the horror that is Docker, it is a wonder that you dare propose to install an entire OS only to run a Matrix server. ;)

                                                                                                                            1. 3

                                                                                                                              i’m not completely sure which parts of you comment are sarcasm :)

                                                                                                                        2. 0

                                                                                                                          Your list there has lots of tools with overlapping functionality, seems like pointless redundancy. Just pick flask OR django. Just pick python3 or node, just pick docker or vagrant, make a choice, remove useless and redundant things.

                                                                                                                          1. 3

                                                                                                                            We have some Django applications and we have some Flask applications. They have different lineages. One we forked and one we made ourselves.

                                                                                                                        3. 6

                                                                                                                          Alternatively you install it using the Docker as described here.

                                                                                                                          1. 31

                                                                                                                            I think it’s kinda sad that the solution to “control your own toots” is “give up control of your computer and install this giant blob of software”.

                                                                                                                            1. 9

                                                                                                                              Piling another forty years of hexadecimal Unix sludge on top of forty years of slightly different hexadecimal Unix sludge to improve our ability to ship software artifacts … it’s an aesthetic nightmare. But I don’t fully understand what our alternatives are.

                                                                                                                              I’ve never been happier to be out of the business of having to think about this in anything but the most cursory detail.

                                                                                                                              1. 11

                                                                                                                                I mean how is that different from running any binary at the end of the day. Unless you’re compiling everything from scratch on the machine starting from the kernel. Running Mastodon from Docker is really no different. And it’s not like anybody is stopping you from either making your own Dockerfile, or just setting up directly on your machine by hand. The original complaint was that it’s too much work, and if that’s a case you have a simple packaged solution. If you don’t like it then roll up the sleeves and do it by hand. I really don’t see the problem here I’m afraid.

                                                                                                                                1. 11

                                                                                                                                  “It’s too much work” is a problem.

                                                                                                                                  1. 5

                                                                                                                                    Unless you’re compiling everything from scratch on the machine starting from the kernel

                                                                                                                                    I use NixOS. I have a set of keys that I set as trusted for signature verification of binaries. The binaries are a cache of the build derivation, so I could theoretically build the software from scratch, if I wanted to, or to verify that the binaries are the same as the cached versions.

                                                                                                                                    1. 2

                                                                                                                                      Right, but if you feel strongly about that then you can make your own Dockerfile from source. The discussion is regarding whether there’s a simple way to get an instance up and running, and there is.

                                                                                                                                      1. 3

                                                                                                                                        Docker containers raise a lot of questions though, even if you use a Dockerfile:

                                                                                                                                        • What am I running?
                                                                                                                                        • Which versions am I running?
                                                                                                                                        • Do the versions have security vulnerabilities?
                                                                                                                                        • Will I be able to build the exact same version in 24 months?

                                                                                                                                        Nix answers these pretty will and fairly accurately.

                                                                                                                                    2. 2

                                                                                                                                      Unless you’re compiling everything from scratch on the machine starting from the kernel.

                                                                                                                                      You mean starting with writing a bootstrapping compiler in assembly, then writing your own full featured compiler and compiling it in the bootstrapping compiler. Then moving on to compiling the kernel.

                                                                                                                                      1. 1

                                                                                                                                        No no, your assembler could be compromised ;)

                                                                                                                                        Better write raw machine code directly onto the disk. Using, perhaps, a magnetized needle and a steady hand, or maybe a butterfly.

                                                                                                                                        1. 2

                                                                                                                                          My bootstrapping concept was having the device boot a program from ROM that takes in the user-supplied, initial program via I/O into RAM. Then passes execution to it. You enter the binary through one of those Morse code things with four buttons: 0, 1, backspace, and enter. Begins executing on enter.

                                                                                                                                          Gotta input the keyboard driver next in binary to use a keyboard. Then the display driver blind using the keyboard. Then storage driver to save things. Then, the OS and other components. ;)

                                                                                                                                        2. 1

                                                                                                                                          If I deploy three Go apps on top of a bare OS (picked Go since it has static binaries), and the Nginx server in front of all 3 of them uses OpenSSL, then I have one OpenSSL to patch whenever the inevitable CVE rolls around. If I deploy three Docker container apps on top of a bare OS, now I have four OpenSSLs to patch - three in the containers and one in my base OS. This complexity balloons very quickly which is terrible for user control. Hell, I have so little control over my one operating system that I had to carefully write a custom tool just to make sure I didn’t miss logfile lines in batch summaries created by cron. How am I supposed to manage four? And three with radically different tooling and methodology to boot.

                                                                                                                                          And Docker upstream, AFAIK, has provided nothing to help with the security problem which is probably why known security vulnerabilities in Docker images are rampant. If they have I would like to know because if it’s decent I would switch to it immediately. See this blog post for more about this problem (especially including links) and how we “solved” it in pump.io (spoiler: it’s a giant hack).

                                                                                                                                          1. 3

                                                                                                                                            That’s not how any of this works. You package the bare minimum needed to run the app in the Docker container, then you front all your containers with a single Nginx server that handles SSL. Meanwhile, there are plenty of great tools, like Dokku for managing Docker based infrastructure. Here’s how you provision a server using Let’s Encrypt with Dokku:

                                                                                                                                            sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
                                                                                                                                            okku letsencrypt:auto-renew
                                                                                                                                            

                                                                                                                                            viewing logs isn’t rocker science either:

                                                                                                                                            dokku logs myapp
                                                                                                                                            
                                                                                                                                            1. 1

                                                                                                                                              OK, so OpenSSL was a bad example. Fair enough. But I think my point still stands - you’ll tend to have at least some duplicate libraries across Docker containers. There’s tooling around managing security vulnerabilities in language-level dependencies; see for example Snyk. But Docker imports the entire native package manager into the “static binary” and I don’t know of any tooling that can track problems in Docker images like that. I guess I could use Clair through Quay but… I don’t know. This doesn’t feel like as nice of a solution or as polished somehow. As an image maintainer I’ve added a big manual burden keeping up with native security updates in addition to those my application actually directly needs, when normally I could rely on admins to do that, probably with lots of automation.

                                                                                                                                              1. 3

                                                                                                                                                you’ll tend to have at least some duplicate libraries across Docker containers

                                                                                                                                                That is literally the entire point. Application dependencies must be separate from one another, because even on a tight-knit team keeping n applications in perfect lockstep is impossible.

                                                                                                                                                1. 1

                                                                                                                                                  OS dependencies are different than application dependencies. I can apply a libc patch on my Debian server with no worry because I know Debian works hard to create a stable base server environment. That’s different than application dependencies, where two applications are much more likely to require conflicting versions of libraries.

                                                                                                                                                  Now, I run most of my stuff on a single server so I’m very used to a heterogeneous environment. Maybe that’s biasing me against Docker. But isn’t that the usecase we’re discussing here anyway? How someone with just a hobbyist server can run Mastodon?

                                                                                                                                                  Thinking about this more I feel like a big part of what bothers me about Docker, and therefore about Clair, is that there’s no package manifest. Dockerfile does not count, because that’s not actually a package manifest, it’s just a list of commands. I can’t e.g. build a lockfile format on top of that, which is what tools like Snyk analyze. Clair is the equivalent of having to run npm install and then go trawling through node_modules looking for known vulnerable code instead of just looking at the lockfile. More broadly, because Docker lacks any notion of a package manifest, it seems to me that while Docker images are immutable once built, the build process that leads you there cannot be made deterministic. This is what makes it hard to keep track of the stuff inside them. I will have to think about this more - as I write this comment I’m wondering if my complaints about duplicated libraries and tracking security there is an instance of the XY problem or if they really are separate things in my mind.

                                                                                                                                                  Maybe I am looking for something like Nix or Guix inside a Docker container. Guix at least can export Docker containers; I suppose I should look into that.

                                                                                                                                                  1. 2

                                                                                                                                                    OS dependencies are different than application dependencies.

                                                                                                                                                    Yes, agreed.

                                                                                                                                                    Thinking about this more I feel like a big part of what bothers me about Docker, and therefore about Clair, is that there’s no package manifest. Dockerfile does not count, because that’s not actually a package manifest, it’s just a list of commands. I can’t e.g. build a lockfile format on top of that, which is what tools like Snyk analyze.

                                                                                                                                                    You don’t need a container to tell you these things. Application dependencies can be checked for exploits straight from the code repo, i.e. brakeman. Both the Gemfile.lock and yarn.lock are available from the root of the repo.

                                                                                                                                                    The container artifacts are most like built automatically for every merge to master, and that entails doing a full system update from the apt repository. So in reality, while not as deterministic as the lockfiles, the system deps in a container are likely to be significantly fresher than a regular server environment.

                                                                                                                                                2. 1

                                                                                                                                                  You’d want to track security vulnerabilities outside your images though. You’d do it at dev time, and update your Dockerfile with updated dependencies when you publish the application. Think of Docker as just a packaging mechanism. It’s same as making an uberjar on the JVM. You package all your code into a container, and run the container. When you want to make updates, you blow the old one away and run a new one.

                                                                                                                                          2. 4

                                                                                                                                            I have only rarely used Docker, and am certainly no booster, so keep that in mind as I ask this.

                                                                                                                                            From the perspective of “install this giant blob of software”, do you see a docker deployment being that different from a single large binary? Particularly the notion of the control that you “give up”, how does that differ between Docker and $ALTERNATIVE?

                                                                                                                                            1. 14

                                                                                                                                              Ideally one would choose door number three, something not so large and inauditable. The complaint is not literally about Docker, but the circumstances which have resulted in docker being the most viable deployment option.

                                                                                                                                            2. 2

                                                                                                                                              You have the dockerfile and can reconstruct. You haven’t given up control.

                                                                                                                                              1. 5

                                                                                                                                                Is there a youtube video I can watch of somebody building a mastodon docker image from scratch?

                                                                                                                                                1. 1

                                                                                                                                                  I do not know of one.

                                                                                                                                          3. 3

                                                                                                                                            I totally agree as well, and I wish authors would s/Mastodon/Fediverse/ in their articles. As others have noted, Pieroma is another good choice and others are getting into the game - NextCloud added fediverse node support in their most recent release as a for-instance.

                                                                                                                                            I tried running my own instance for several months, and it eventually blew up. In addition to the large set of dependencies, the system is overall quite complex. I had several devs from the project look at my instance, and the only thing they could say is it was a “back-end problem” (My instance had stopped getting new posts).

                                                                                                                                            I gave up and am now using somebody else’s :) I love the fediverse though, it’s a fascinating place.

                                                                                                                                            1. 4

                                                                                                                                              I just use the official Docker containers. The tootsuite/mastodon container can be used to launch web, streaming, sidekiq and even database migrations. Then you just need an nginx container, a redis container, a postgres container and an optional elastic search container. I run it all on a 2GB/1vCPU Vultr node (with the NJ data center block store because you will need a lot of space) and it works fairly well (I only have ~10 users; small private server).

                                                                                                                                              In the past I would agree with out (and it’s the reason I didn’t try out Diaspora years ago when it came out), but containers have made it easier. I do realize they both solve and cause problems and by no means think they’re the end all of tech, but they do make running stuff like this a lot easier.

                                                                                                                                              If anyone wants to find me, I’m @djsumdog@hitchhiker.social

                                                                                                                                              1. 2

                                                                                                                                                Given that there’s a space for your Twitter handle, i wish Lobste.rs had a Mastodon slot as well :)

                                                                                                                                              2. 2

                                                                                                                                                Wait, you’re also forgetting systemd to keep all those process humming… :)

                                                                                                                                                You’re right that this is clearly too much: I have run such systems for work (Rails’ pretty common), but would probably not do that for fun. I am amazed, and thankful, for the people who volunteer the effort to run all this on their week-ends.

                                                                                                                                                Pleroma does look simpler… If I really wanted to run my own instance, I’d look in that direction. ¯_(ツ)_/¯

                                                                                                                                                1. 0

                                                                                                                                                  I’m waiting for urbit.org to reach useability. Which I expect for my arbitrary feeling of useability to come about late this year. Then the issue is coming up to speed on a new language and integrated network, OS, build system.

                                                                                                                                                  1. 2

                                                                                                                                                    Urbit is apparently creating a feudal society. (Should note that I haven’t really dug into that thread for several years and am mostly taking @pushcx at his word.)

                                                                                                                                                    1. 1

                                                                                                                                                      The feudal society meme is just not true, and, BTW, Yarvin is no longer associated with Urbit. https://urbit.org/primer/

                                                                                                                                                  2. 1

                                                                                                                                                    I would love to have(make) a solution that could be used locally with sqlite and in aws with lambda, api gateway and dynamodb. That would allow scaling cost and privacy/controll.

                                                                                                                                                    1. 3

                                                                                                                                                      https://github.com/deoxxa/don is sort of in that direction (single binary, single file sqlite database).

                                                                                                                                                  1. 3

                                                                                                                                                    nitpick:

                                                                                                                                                    Note: even though it originally came from an acronym, Tor is not spelled “TOR”. Only the first letter is capitalized. In fact, we can usually spot people who haven’t read any of our website (and have instead learned everything they know about Tor from news articles) by the fact that they spell it wrong.

                                                                                                                                                    https://www.torproject.org/docs/faq.html.en#WhyCalledTor

                                                                                                                                                    1. 2

                                                                                                                                                      The article uses both capitalizations. Maybe it was written by multiple people? Poorly edited anyway.

                                                                                                                                                    1. 2

                                                                                                                                                      This is really interesting work. Since it’s from 2016 and the authors have implemented it in FreeBSD as a proof of concept, is there any news on this topic? @lattera: would this fit in HardenedBSD?

                                                                                                                                                      1. 2

                                                                                                                                                        I would definitely not mind reviewing a patch from the community providing support for this in HardenedBSD. :)

                                                                                                                                                        1. 4

                                                                                                                                                          Putting a patch together wouldn’t be too bad and largely be self-contained, but I’ll take a moment to say where the warts are in terms of the implementation and what I’d have to do to get a patch ready.

                                                                                                                                                          So to step up a level, one of the core things here is that as a thread is executing it can change the address space in which it is operating. With a single-threaded process, like nginx, this means you can have an address space per protection domain and the semantics make sense.

                                                                                                                                                          Things get a little stranger when you have multiple threads, because under this model the process no longer has one operating space that all threads are operating in. Instead, threads may share address spaces, or not, depending on the intraprocess protection domain they execute in. This is the thing that makes the changeset the largest, because the vmspace now hangs out of the thread instead of the process. This required something akin to s/p->p_vmspace/td->td_vmspace/g all over the code and some work on what happens when threads/processes are created and destroyed. A version of lwcs that only was permitted in single-threaded processes would be more self-contained.

                                                                                                                                                          The semantics of how other threads should be treated when a separate one creates a new lwc was a bit of a heated topic for us while we were working on the paper. I can go into the details, but the short version is that I think the thread semantics should be improved so it’s easier to program and reason about.

                                                                                                                                                          The other thing is I didn’t implement the signal propagation. I think this is pretty easy though.

                                                                                                                                                          1. 1

                                                                                                                                                            Friendly ping. :)

                                                                                                                                                            Can you email the HardenedBSD dev team (dev@hardenedbsd.org)? I’d like to help bring this work into HardenedBSD.

                                                                                                                                                            And, given that the university isn’t too far from where I live/work, I’d be happy to stop on by for a conversation/brainstorm session. :)

                                                                                                                                                            1. 1

                                                                                                                                                              I’ve sent off an email, thanks.

                                                                                                                                                          2. 3

                                                                                                                                                            Also, the university that this came out of isn’t too far from where I live. I could probably ping the people who wrote this and see about readying a patch for HardenedBSD.

                                                                                                                                                        1. 4

                                                                                                                                                          Actual link contained in the sub-tweet. It barely says anything and it’s nothing offical from Mozilla.

                                                                                                                                                          As I, @friendlysock and others have implored others for a while: please do not link news. That’s not what this site is about, nor what it should be about. Other sites do news already. Lobste.rs shouldn’t. And it’s very rare that Twitter submissions are worthy.

                                                                                                                                                          Here’s the full content of the article linked in the article that you eventually get to by following the submission (emphasis added). It’s extremely light. The quote making up the title isn’t even from someone at Mozilla. If what the title of the submission says is true, it would be helpful to see it from Mozilla itself, not an unsourced blurb from a writer on Fast Company.

                                                                                                                                                          When Facebook users learned last March that the social media giant had given their sensitive information to political-data firm Cambridge Analytica, Mozilla (parent company of the security-focused browser Firefox) reacted fast: Within eight hours, the product team had built a browser extension called the Facebook Container. The plug-in, now the most popular browser extension Mozilla has ever built (1.5 million downloads and 500,000 monthly active users), prevents Facebook from trailing its users around the internet. Firefox Monitor, a service Mozilla launched in September, uses your email address to determine whether your personal info has been compromised in a breach. By summer 2019, the Firefox browser will also block, by default, all cross-site third-party trackers, strengthening privacy without your having to do a thing (unlike Firefox’s biggest competitor, Google Chrome). “We want to make it simple for people to create walls around data that’s important to them,” says Denelle Dixon, Mozilla’s COO.

                                                                                                                                                          1. 1

                                                                                                                                                            Actual link contained in the sub-tweet. It barely says anything and it’s nothing offical from Mozilla.

                                                                                                                                                            Although Jen Simmons, the one making the quote is Designer Advocate at Mozilla.

                                                                                                                                                            As I, @friendlysock and others have implored others for a while: please do not link news. That’s not what this site is about, nor what it should be about. Other sites do news already. Lobste.rs shouldn’t.

                                                                                                                                                            Fair enough. Maybe we should add that to the story submission guidelines.

                                                                                                                                                            1. 1

                                                                                                                                                              Although Jen Simmons, the one making the quote is Designer Advocate at Mozilla.

                                                                                                                                                              Maybe she’s being coy since she’s only quoting the article. Regardless, it’s not an official statement so it’s all conjecture at this point.

                                                                                                                                                              Maybe we should add that to the story submission guidelines.

                                                                                                                                                              This is a good point. We should talk about in the IRC channel.

                                                                                                                                                          1. 20

                                                                                                                                                            I think this is huge. It reminds me of the early days of Firefox (back then still known as Phoenix) in a world where IE6 and pop-up ads dominated. At launch IE6 was really the best and most innovative browser of it’s time (IMHO). But after IE6 had beaten Netscape, Microsoft stopped putting money in IE development and the situation got worse over time. It was Phoenix with, among other things, a pop-up blocker that was on by default that brought down Internet Explorers hegemony.

                                                                                                                                                            Today, with Chrome being dominant the situation is different because Google is still innovating Chrome at light speed. The one and only Achilles heel to beat this giant is by attacking their business model, which is to enable ad blocking by default. I expect this is something people want, just like pop-up blockers back in the days. Google will never be able to lead, or even follow in this direction without changing their business model.

                                                                                                                                                            Unfortunately, Mozilla’s own business model also heavily relies on selling ads, albeit indirectly. According to this statement from an independent audit report:

                                                                                                                                                            Note 10 - Concentrations of Risk:

                                                                                                                                                            Mozilla has entered into contracts with search engine providers for royalties which expire through November 2020. Approximately 93% and 94% of Mozilla’s royalty revenues were derived from these contracts for 2017 and 2016, respectively, with receivables from these contracts representing approximately 75% and 79% of the December 31, 2017 and 2016 outstanding receivables.

                                                                                                                                                            In other words, $539 Million, which is 93% of their total revenue, comes from companies that have selling ads as their business model (Baidu, Google, Yahoo and Yandex).

                                                                                                                                                            I really hope Mozilla will be able to change this revenue stream to better align with their mission[1]. They have been trying to diversify their revenue since 2014 and although they might not be as dependent on Google as they once were, they’re still almost fully dependent on ads.

                                                                                                                                                            Oh, and yeah, of course simply making a better browser than Chrome would also help ;)

                                                                                                                                                            Background:

                                                                                                                                                            [1] https://www.mozilla.org/en-US/mission/ “An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent.”

                                                                                                                                                            1. 11

                                                                                                                                                              I really hope Mozilla will be able to change this revenue stream to better align with their mission[1]. They have been trying to diversify their revenue since 2014 and although they might not be as dependent on Google as they once were, they’re still almost fully dependent on ads.

                                                                                                                                                              This is the million dollar question / problem. The world does not run on fairy dust and good intentions. Figuring out HOW to better align with their mission is a decidedly non trivial problem, and I’d wager that were it not they’d have done this already, because everybody and their uncle likes to carp about Mozilla.

                                                                                                                                                              1. 6

                                                                                                                                                                I think they should introduce a pro-version with a different firefox icon for 5$/month, I would buy it.

                                                                                                                                                                1. 2

                                                                                                                                                                  So… they’d need to find ~10 million people like you who will pay for an icon in order to be able to replace their ad-related revenue stream. I doubt that’s going to happen, or that they’re even going to have any viable revenue stream out of this.

                                                                                                                                                                  As the amount in question is $500M/year, Mozilla’s problem is similar to the situation that large corporations face, where they are unable to enter many markets because they are just too small. Mozilla needs to find large revenue streams, and it’s impossible for them to combine lots of tiny revenue streams (by doing things like you suggested) because the overhead would be too high.

                                                                                                                                                                  1. 1

                                                                                                                                                                    I don’t agree. First, I don’t understand why browser development (since we are talking about firefox) should cost $500M/year. Second: Diversification. Icon was just one proposal, another can be payed VPN, another can be enterprise features proposed in this thread. (Hated) pocket integration is another one. Mozilla in some sense reminds me of Europe that for 20 years cannot diversify itself from the russian gas (money coming from the ads companies).

                                                                                                                                                                    1. 2

                                                                                                                                                                      My point is that you can’t replace $500M/year with tiny revenue streams like that. You’d need hundreds of them, and it’s simply unmanageable.

                                                                                                                                                                      Why it costs so much: it’s probably because there’s a lot more than just one browser. Whether there really need to be so many projects is another question, I don’t really know the answer to that.

                                                                                                                                                                      1. 1

                                                                                                                                                                        One problem I expect they have is that shrinking costs is really hard, and growing them really easy. The vast majority of it is probably salaries. Trying to shrink from $500M/year to $50M/year (say) would mean getting rid of roughly 9/10 employees!

                                                                                                                                                                        To do this successfully you’d need to retain quality employees, and make sure that the administrative staff shrunk as much as (or more than) the engineering staff. Not something that is easy to do while firing 90% of your staff, going on a very long hiring freeze, or so on.

                                                                                                                                                                    2. 2

                                                                                                                                                                      I said this a long time ago. Make some privacy-focused, enterprise offerings on top of it with good service. Additionally, an auditability-focused version supporting logging, messaging, etc. Enterprises buy lots of that stuff, too.

                                                                                                                                                                      1. 11

                                                                                                                                                                        We’re getting there. There was already a VPN service that we experimented with at the end of last year (I believe signups are closed now) and there are other ideas in the works as well.

                                                                                                                                                                        The hard part is figuring out what is acceptable to charge for. The last thing we would want is a world where users need to pay for increased privacy. Enterprise offerings could certainly fit the bill.

                                                                                                                                                                        1. 2

                                                                                                                                                                          Have you considered EU sponsoring?

                                                                                                                                                                          1. 2

                                                                                                                                                                            I am very far removed from this process, all I know is what’s been shared by leadership internally :). But from what I can tell there have been a ton of ideas and the process has been very thorough. I’d be surprised if this hasn’t been considered.

                                                                                                                                                                          2. 1

                                                                                                                                                                            ahal, start with changing the icon ;) Once the infra for two repos is set, you will see where it brings you.

                                                                                                                                                                        2. 1

                                                                                                                                                                          As would I, however you don’t have to look far at all to see that this amounts to good intentions. Sure, a few of us would pony up, but the VAST majority of users simply will not. They want a free lunch.

                                                                                                                                                                      2. 4

                                                                                                                                                                        Yeah, just think of the speed benefit they could brag about by simply blocking all ads by default too. Privacy is great by itself, but imagine the wow of being 2x or 3x faster than the “fast” google chrome with such a simple change on top of it!

                                                                                                                                                                      1. 10

                                                                                                                                                                        Is this conceptually different from Delta Chat that has been around for quite some time now?

                                                                                                                                                                        1. 5

                                                                                                                                                                          The COI developers promise that Delta Chat Core will become COI compliant.

                                                                                                                                                                          As a client developer, you have several options:

                                                                                                                                                                          • Communicate with the email server using an IMAP library for your preferred language & platform – or even communicate directly when you feel adventurous.
                                                                                                                                                                          • Base your work on Delta Chat Core, an MPL licensed library that abstracts away IMAP communication and that will become COI compliant.
                                                                                                                                                                          • Base your work on an existing COI compatible app, like the cross-platform OX Talk app.

                                                                                                                                                                          From https://confluence-public.open-xchange.com/display/CoiW/COI+for+Beginners#COIforBeginners-Whatyouneed

                                                                                                                                                                          1. 5

                                                                                                                                                                            It looks like this may build upon Delta Chat - similarly working with any IMAP server but also checking for a capability named COI to support more features with server-side support. This was the only direct mention I found. I checked a few peoples’ names and I’m not completely certain about cooperation here?

                                                                                                                                                                            I wonder if they were aware “COI” was already a term in use specifically in email contexts, though maybe the scopes are different enough they’d be hard to confuse…

                                                                                                                                                                          1. 2

                                                                                                                                                                            Interesting, I like your point about realism. I’ve been writing some tests that use a database mock instead of a real database for performance reasons, even though the only databases supported in that codebase are memory based. It could be argued that if you want to unit test a library, you don’t want to test it’s storage layer. Of course the rules blend when some business logic relied on by the library is enforced in the storage layer (for example uniqueness).

                                                                                                                                                                            1. 1

                                                                                                                                                                              From https://eprint.iacr.org/2018/1173.pdf:

                                                                                                                                                                              We have tested nine fully patched implementations of various RSA-based security protocols (OpenSSL, Amazon s2n, MbedTLS, Apple CoreTLS, Mozilla NSS, WolfSSL, GnuTLS, BearSSL and BoringSSL). While all of these implementations attempt to protect against microarchitectural and timing side channel attacks, we describe new side channel attack techniques which overcome the padding oracle countermeasures. Notably, out of the nine evaluated implementations, only the last two (BearSSL and BoringSSL) could not be successfully attacked by our new techniques.

                                                                                                                                                                              Does anybody know if LibreSSL is vulnerable?

                                                                                                                                                                              1. 3

                                                                                                                                                                                They didn’t test it as it did not support TLS1. 3 at the time of testing (not sure if it does now).

                                                                                                                                                                                1. 1

                                                                                                                                                                                  But BearSSL doesn’t support tls 1.3 either?

                                                                                                                                                                                2. 2

                                                                                                                                                                                  I think so, though I’m not certain exactly where or when it was fixed.

                                                                                                                                                                                  1. 2

                                                                                                                                                                                    That’s exactly what I thought when I read that list.