1. 3

    This is about STREAMS, a more general full-duplex interface to pipes and UNIX Domain sockets in the BSD socket interface. The STREAMS work ended up in SystemV and later Solaris. You can clearly see the influence of Plan9 by the way file descriptors are passed and protocols are abstracted in the file system.

    I think it’s too bad that Node.js streams are only a shallow simplex version of this work (Joyent built and maintained SmartOS, which is based on Solaris and ships with the STREAMS API so I would expect them to be famiilar with this design).

    1. 3

      I don’t think there was any direct inspiration drawn from STREAMS in the Node Streams APIs (of which there have now been at least three fairly different versions also).

      There are aspects of STREAMS that are somewhat interesting, but the programming model is, to be charitable, cumbersome. It also hasn’t stood up in the era of increased expectations around networking performance; subsequent improvements to networking began with fast path bypasses of the generic STREAMS underpinning.

      1. 1

        I don’t think there was any direct inspiration drawn from STREAMS in the Node Streams APIs (of which there have now been at least three fairly different versions also).

        Thanks for chiming in!

        There are aspects of STREAMS that are somewhat interesting, but the programming model is, to be charitable, cumbersome.

        Looking at the latest version of the STREAMS Programming Guide (which includes more than a decade of development compared to what this paper from 1990 describes) I think it definitely looks over-engineered for a lot of use cases.

        As a side note, since you work on Illumos, do you know why the STREAMS Programming Guide is not listed under https://illumos.org/books? Is STREAMS in Solaris also slowly rotting away?

        1. 1

          I think the main reason it’s missing is that we don’t, as far as I know, have the docbook source code for that one. The books we have a source licence for are at: https://github.com/illumos/illumos-docbooks

          I can’t speak for the Solaris folks, but in illumos I don’t think we’ve had any attempts to improve or increase the use of STREAMS. A lot of terminal handling is still STREAMS-heavy, but that’s as much a historical artefact as anything else. If someone feels the need to iterate on that in a major way, I’m not sure that it’d still be STREAMS afterwards.

      2. 1

        You can clearly see the influence of Plan9 by the way file descriptors are passed and protocols are abstracted in the file system.

        Could you share an example that demonstrates this?

        1. 1

          The ease to pass file descriptors with sendfile() and recvfile() instead of all the CMSG_* stuff with Berkely sockets. The generalized mounting. The way addresses used in ipcopen() and ipccreat() are delimited like namespaces in a file-system and abstract whether a service is local or remote. Plan 9 has based their kernel structure for networks on an earlier version of the work presented in this paper and Dave Presotto worked both on STREAMS and Plan 9: “The kernel plumbing used to build Plan 9 communications channels is called streams”.

      1. 3

        Note that U2F keys are a new OpenSSH key type, so the server must support it too.

        Anyone know why this would be the case? I was hoping the keypair could just be presented to the server as a regular ECDSA key.

        1. 4

          It was not possible to enable U2F keys using the existing SSH ECDSA key format because, despite U2F devices using ECDSA for signatures, the signature format differs from the plain signatures used in SSH by including some additional fields.

          1. 1

            I noticed that, but I didn’t understand why those additional fields are there, or why it would cause a problem.

            My best guess now, after seeing ClashTheBunny’s reply, is that 1) the additional fields allow for at least the signature counter, and 2) the existing signature validation mechanism doesn’t tolerate unknown, additional fields (which would be a reasonable choice, really.)

            1. 2

              In most crypto schemes the thing that is signed is actually a digest of some more complex byte structure. That they don’t match here is not surprising. U2F for example requires adding origin (domain name) and a challenge. Even if the required data were the same the binary encoding could be different and that alone would make the signatures incompatible.

          2. 3

            I get the question now.

            I wonder if it is to “promise” that the secret key material is on a hardware store and not a software store?

            1. 4

              It does seem that there is attestation PROTOCOL.u2f:

              U2F also offers a number of features that are attractive in the context of SSH authentication. They can be configured to require indication of “user presence” for each signature operation (typically achieved by requiring the user touch the key). They also offer an attestation mechanism at key enrollment time that can be used to prove that a given key is backed by hardware. Finally the signature format includes a monotonic signature counter that can be used (at scale) to detect concurrent use of a private key, should it be extracted from hardware.

          1. 3

            This one from 2.5 Distraction 2: minimizing privilege really made me think, especially in relation to the benefits of sandboxing techniques like pledge(2):

            I have become convinced that this “principle of least privilege” is fundamentally wrong. Minimizing privilege might reduce the damage done by some security holes but almost never fixes the holes. Minimizing privilege is not the same as minimizing the amount of trusted code, does not have the same benefits as minimizing the amount of trusted code, and does not move us any closer to a secure computer system.

            If I understand correctly then his argument is that the principle of least privilege is no more than a security mitigation, limiting collateral damage at best. Whereas proper separation and having a bug-free TCB are the only security solutions. Or in other words an unprivileged process (from the operating system point of view) that handles all incoming mail for all users on the system still breaks the users security requirements once exploited since the process has access to all incoming mail of all users. If the same program with the same security vulnerability was configured to only handle one incoming message and then exit (and thus be forked for every incoming message), it could never break the security requirement of any user, even when exploited.

            1. 2

              That’s a very interesting observation, and I feel sort of reveals a difference in the underlying assumptions: do you expect that There’s Always One More Bug, and so the mitigations are useful, or do you expect you can make a truly secure core kernel of your system and design accordingly?

              1. 2

                Whereas proper separation and having a bug-free TCB are the only security solutions

                That is how the strongest solutions implement least privilege. The separation architecture with whitelisted communications starts with no privilege before giving just a few for IPC. It’s doing the same thing. From there, bug-free TCB’s are hard enough that high-assurance security reduced bug-free to tiniest components (eg separation kernel) that they could. Then, they tried to expand what those capabilities could handle with medium-assurance techniques, like memory safety and covert-channel analysis, handling other components with less impact on security policy.

                In fact, the Least-Privilege, Separation Kernels achieved higher assurance than Bernstein’s designs given their proofs covered all modeled states for both correctness and leak freedom assuming trustworthy hardware. Then, someone found a problem in his before others found thousands of bugs in kernel it ran on. Then, there’s no covert-channel analysis of either qmail or Linux that I know of. Least privilege designs with EAL7 process still wins over his highly-respectable work.

                It lost in long run due to relying on COTS hardware with vulnerabilities and leaks. The assumption was never correct on high-performance, production CPU’s. That was predicted back in 1995, though, after an analysis of Intel CPU’s for leaks called for leak-proof components in processors. Verified processors such as VAMP DLX and Rockwell-Collins’ AAMP7G were created with latter in real-world use. They’re not desktop or server grade, though.

                1. 1

                  I was hoping you would chime in, thanks!

                  In 5.2 Bernstein does mention that his assumption that the CPU does not leak secret information from one process to another, is debatable.

                  1. 2

                    “I am also assuming that the CPU does not leak secret information from one process to another. This assumption is debatable.”

                    It good that you brought it up. I either forgot or sort of glanced over that statement. I wonder what he was thinking when he wrote it. Quite a few people worried about things like prior state of the registers leaking information or CPU just glitching. Was it as little as that or something leaning toward bigger or deeper leaks? We won’t know. Anyway, just add that observation to his security cred. :)

                    Far as non-leaking CPU’s, this pdf is a good one among those that aren’t paywalled. Always remember, too, that much of the “proof” benefit can be gained by precise specifications, detailed reviews, static analysis, and lots of testing. The proof parts are impressive and capable but not necessary if aiming for qmail-like assurance (Medium - stop many attackers). 80/20 rule applies in INFOSEC, too, if not aiming for perfection.

              1. 2

                Actually, the linked article: https://blog.cloudflare.com/warp-technical-challenges/ offers more in-depth description of their hurdles and how they overcame them.

                1. 24

                  Fortunately, WireGuard has a three byte block in its header which is not currently used by other clients. We decided to put our identifier in this region

                  I believe the technical term for referring to this is “a dick move”. First CF decides to just go off and make their one wireguard impl rather than contributing to the existing opensource ones now they’re not even following the protocol? That’s not very neighbourly. I would have thought that CF would have had to deal with the problems of protocol ossification enough to know that they shouldn’t do this.

                  1. 4

                    I want to share my thought process here. At first, I sneered “If their 1.1.1.1 app (client) never connects to any server other than cloudflare, what’s it matter?” Please note that I prefer to use clients that use protocols to connect to servers of my choosing. I like to participate on the Internet rather than merely consume…

                    But then I thought about how the FTP protocol changed over time as network position of FTP clients changed over time: first on mostly single-activity-at-a-time machines, to multi-user multi-process machines, to machines on networks that were not professionally managed, to machines behind consumer NAT devices. Some of you just know what I’m talking about–the history of ftp clients, ftp daemons, router manufacturers, and ISPs, and what a mess it was. If you don’t, I guess this might be a legit starting poing? https://duckduckgo.com/?q=FTP+PASV+NAT

                    But azdle’s point is spot on. The CF engineers surely do know that whole sordid history–and the histories of http, of browsers, of transparent http proxies, etc. Sticking to an open protocol and contributing to it as needed is absolutely the right answer and CF surely knows better–even if their shiny new thing is merely an “app”.

                  2. 3

                    I’m a bit sad that, from a technical point of view, there seems to be an influx of lower quality articles posted on Lobste.rs for the last couple of months. I’m seeing more and more web stuff and commercial blog postings that are only light on technical details. This is a good example of what I think should have been posted instead of the OP.

                  1. 24

                    I see too many people rolling PHP-FPM only to show an IP address to the client. So, I wanted to share a simpler method which, I hope, can save you some time.

                    1. 3

                      This seems very elegant, but just to be thorough are there any drawbacks/tradeoffs?

                      1. 4

                        The same T&Cs apply as when using nginx for standard stuff. This will/might be wrong if this nginx is behind another nginx, then you should look at X_FORWARDED_FOR (or whatever it’s called exactly).

                        1. 2

                          Beware if you use another public facing server in front of nginx. For example, if you have a reverse proxy (HAproxy for example), then the variable $remote_addr can represent the IP address of the proxy, not the initial HTTP client.

                          1. 4

                            Have a look at the realip module that allows Nginx to set the remote address based on a header set by the frontend proxy provided it’s one you have decided to trust that it’s setting correct headers.

                            Doing this over a custom solution in the application has the advantage that all remote address based features continue to work unaltered. Like geoip detection or logging addresses to web log files using built-in standard formats

                        2. 2

                          Yes this method also has my preference and we use that for years now. Years ago we used to use php (without php-fpm) for this, more or less like this:

                          <?php
                          echo $_SERVER['REMOTE_ADDR'] . PHP_EOL;
                          

                          But I was wondering: do you have suggestions for making it output both IPv4 and IPv6 addresses (like https://ip6.nl and others do) without adding additional complexity/dependencies like php (preferably with stock nginx or apache).

                          1. 4

                            To show both IPv4 and IPv6, the client needs to make two separate requests, to two separate domains that are configured differently, one with only an A record and one with only an AAAA record. Any given HTTP request is only going to be coming in on one or the other IP version.

                            ip6.nl makes XHR requests to https://4only.ip6.nl/myip.plp and https://6only.ip6.nl/myip.plp and displays the results on the page, again with Javascript. While those servers could very well be running the nginx config in the linked article, the ability to show both on the same page is much more complicated, tech-wise.

                            1. 2

                              You might be able to do it with redirects. Have the IPv4 server redirect to the IPv6 server with ?v4=a.b.c.d, and vice versa. Both servers would display both addresses once available.

                              It falls apart if you only have one type of address, since the redirect would be broken, but there’s probably a way around that. Maybe include the single address in the body of the 303, so if the redirect fails to connect you still have the initial IP address you used?

                              1. 3

                                The case where the caller can only connect on one protocol is probably very, very common still.

                            2. 3

                              But I was wondering: do you have suggestions for making it output both IPv4 and IPv6 addresses (like https://ip6.nl and others do) without adding additional complexity/dependencies like php (preferably with stock nginx or apache).

                              The tcp/ip stack of the client decides whether to try to connect using v4 or v6 first. I’ve added two extra dns entries, one with only a v4 address, and one with only a v6 address atop of one that has both: http://ip.netsend.nl http://ip4.netsend.nl http://ip6.netsend.nl

                            3. 2

                              Nice trick! Thanks!

                              However, you could add links to the relevant nginx-pages to your blog post as well.

                            1. 4

                              There is also vis, comparison of vis and kakoune here.

                              1. 6

                                One major difference is that kakoune is being actively developed and has an active community around it, while vis seems to have gone a bit quiet.

                                I tried vis in the past and ran into quite some rough edges; it didn’t seem to be in a state where it’s “done”.

                                1. 1

                                  What kind of “rough edges”?

                                  1. 1

                                    I particularly remember having to force quit it after it became unresponsive, e.g. related to search prompt editing and resizing the terminal.

                                2. 3

                                  I’ve looked into vis yesterday (as well as kakoune) as I’m looking for an editor that supports auto-complete and can jump to definitions like ctags in vi(1). My main reason is to make exploring new and possibly big code bases easier.

                                  It wasn’t clear to me if these features are already supported by vis (I saw mentions of ctags in the issues on github, but nothing concrete). I’ve also seen vis-complete, but it wasn’t quite clear to me how to set this up to span all source code in a repository. Are these things you have working in your vis setup?

                                  1. 1

                                    There is apparently a plugin for ctags in vis, though I’ve never used them, nor have I used autocomplete, sorry.

                                    1. 1

                                      kak-lsp will give you both features (and more, including floating docs, param documentation, etc)

                                  1. 1

                                    Oh no, not again.

                                    It’s clearly time exim was sent to the great mailer-daemon in the sky. How many RCE CVE’s in the last two years? Too many.

                                    Sadly there don’t seem to be any open source SMTP servers written in memory safe langauges around. Unless I’ve missed one?

                                    1. 1

                                      Fortunately there are SMTP servers with a proper design that greatly reduce the severity of the effects of memory corruption.

                                      1. 2

                                        I disagree that those 2 examples are good solutions: Postfix’s configuration is even less readable than Exim’s, and OpenSMTPd is really under-documented and looks much more trouble to run on Linux than it’s worth. Exim is still the least bad of the bunch.

                                        1. 1

                                          All of them should be run in a container (or jail/chroot) if not a VM (QubesOS). Furthermore, we need to get rid of root.

                                          **edit, oh jebus. It looks like ASN.1 parsing strikes again. The most profitable back door in the history of computers.

                                      1. 1

                                        One thing that I don’t think this cookie thing solves is CSRF when it comes to POSTs done by unauthenticated users.

                                        If you have a comment form on your site but no anti-CSRF token in the submit form, then people could just trigger POST requests for your website from theirs.

                                        the anti-CSRF token lets you verify that the form was generated by you, so to speak.

                                        I would be pretty interested in not needing to deal with anti-CSRF tokens on validated forms in general though…

                                        1. 3

                                          That’s what this brings you. Doesn’t it? You can get it NOW by explicitly setting your cookies as SameSite. All major browsers already support the SameSite attribute, but not all set it by default.

                                          1. 2

                                            No? The question being asked here is whether you can get CSRF protection on forms with no required cookies. A mitigation that limits cookies won’t fix that.

                                            1. 4

                                              If the form is anonymous, is CSRF even a thing? If there’s no user to attack, then there’s no harm. At best it seems like nuisance comparable to hotlinking.

                                              1. 1

                                                Kind of, though it’s a bit more work than just what CSRF normally takes.

                                                A typical site that accepts anonymous submissions will use IP address rate limits. A site operator that wants to screw over another site could embed a script that submits a hidden form in an iframe over and over, essentially using the visitors as DoS reflectors. (Obviously, you could make them deny service with just image hotlinking, but having everybody that visits your site automatically submit a question to Stack Overflow in the background would be much more damaging with much fewer visitors).

                                                1. 1

                                                  Not even a nuisance: a feature! Lots of useful things can rely on forms from one site POSTing to another – in fact, even CSRF would be useful if there were some way a browser could warn a user to prevent the attack-mode version of it.

                                                2. 2

                                                  Oh, thank you. I misread that. You can soon get this with fetch-metada though.

                                              2. 2

                                                From a 2017 article by the same author, just set SameSite=lax on the cookies you send to your visitors:

                                                Set-Cookie: sess=abc123; path=/; SameSite=lax
                                                

                                                The OP is that Chrome is about to make this the default for all received cookies in an upcoming release.

                                                1. 2

                                                  Anti-CSRF works by generating a secret when you visit a page. This allows you to present that secret the next time you take an action. This provides trust that you are making the request, not some malicious javascript from a different site (eg, cross site).

                                                  This cookie option allows the cookie to be sent with a request only when the browser is viewing the page, and is not sent for cross site requests. In other words, if a server is getting the cookie, the server must have sent it, the browser supports the option, and the option was set to “Lax,” then there’s no reason for the anti-csrf token.

                                                  1. 1

                                                    You could issue one SameSite cookie to all unauthenticated users and check for that cookie (or one for a logged-in session) when validating the form submission.

                                                  1. 7

                                                    This is great news! Now I can finally recommend OpenBSD to my friends that need to run Apache / PHP / NGINX / Samba :)

                                                    1. 6

                                                      One statistic I’m interested in is the number of submitted stories authored by ppl from the community. I think this is a strong differentiator of Lobste.rs compared to other link aggregation sites.

                                                      1. 3

                                                        There appears to be an increase in them recently, too.

                                                        1. 3

                                                          I had a spot of insomnia and got curious, so I wrote the query. (Reminder I’m happy to run queries, see /about.) The user_is_author boolean was added to stories on 2019-07-30.

                                                          > select extract(year from created_at) as y, extract(month from created_at) as m, sum(user_is_author) as authored, count(*) as "all", round(sum(user_is_author)/count(*)*100) as p
                                                           from stories where created_at >= '2015-07-01' group by 1, 2;
                                                          +------+------+----------+------+------+
                                                          | y    | m    | authored | all  | p    |
                                                          +------+------+----------+------+------+
                                                          | 2015 |    7 |        2 |  608 |    0 |
                                                          | 2015 |    8 |       88 |  541 |   16 |
                                                          | 2015 |    9 |      106 |  549 |   19 |
                                                          | 2015 |   10 |      117 |  737 |   16 |
                                                          | 2015 |   11 |      109 |  791 |   14 |
                                                          | 2015 |   12 |      124 |  761 |   16 |
                                                          | 2016 |    1 |      116 |  972 |   12 |
                                                          | 2016 |    2 |      117 |  849 |   14 |
                                                          | 2016 |    3 |      131 |  736 |   18 |
                                                          | 2016 |    4 |      131 |  739 |   18 |
                                                          | 2016 |    5 |      132 |  786 |   17 |         
                                                          | 2016 |    6 |      157 |  798 |   20 |         
                                                          | 2016 |    7 |      129 |  812 |   16 |                                                          
                                                          | 2016 |    8 |      124 |  797 |   16 |
                                                          | 2016 |    9 |      110 |  731 |   15 |                                                          
                                                          | 2016 |   10 |      115 |  779 |   15 |         
                                                          | 2016 |   11 |      116 |  835 |   14 |  
                                                          | 2016 |   12 |      119 |  852 |   14 |                                                          
                                                          | 2017 |    1 |      152 | 1037 |   15 |       
                                                          | 2017 |    2 |      139 | 1068 |   13 |
                                                          | 2017 |    3 |      182 | 1194 |   15 |  
                                                          | 2017 |    4 |      147 |  947 |   16 |         
                                                          | 2017 |    5 |      160 |  979 |   16 |
                                                          | 2017 |    6 |      175 |  941 |   19 |                                                          
                                                          | 2017 |    7 |      206 | 1109 |   19 |
                                                          | 2017 |    8 |      170 | 1111 |   15 |
                                                          | 2017 |    9 |      202 |  974 |   21 |                                                                                                                                                            
                                                          | 2017 |   10 |      223 |  985 |   23 |                                                          
                                                          | 2017 |   11 |      180 |  924 |   19 |                                                          
                                                          | 2017 |   12 |      165 |  922 |   18 |         
                                                          | 2018 |    1 |      227 |  961 |   24 |                                                          
                                                          | 2018 |    2 |      181 |  846 |   21 |
                                                          | 2018 |    3 |      212 | 1058 |   20 |                                                          
                                                          | 2018 |    4 |      213 |  983 |   22 |         
                                                          | 2018 |    5 |      200 |  982 |   20 |                                                          
                                                          | 2018 |    6 |      188 |  886 |   21 |         
                                                          | 2018 |    7 |      198 | 1017 |   19 |                                                          
                                                          | 2018 |    8 |      207 |  985 |   21 |         
                                                          | 2018 |    9 |      212 |  869 |   24 |                                                                                                                                                            
                                                          | 2018 |   10 |      207 |  918 |   23 |                                                          
                                                          | 2018 |   11 |      221 |  924 |   24 |
                                                          | 2018 |   12 |      225 |  905 |   25 |
                                                          | 2019 |    1 |      218 | 1058 |   21 |
                                                          | 2019 |    2 |      207 |  924 |   22 |
                                                          | 2019 |    3 |      246 |  978 |   25 |
                                                          | 2019 |    4 |      271 |  967 |   28 |
                                                          | 2019 |    5 |      229 |  954 |   24 |
                                                          | 2019 |    6 |      260 |  925 |   28 |
                                                          | 2019 |    7 |       62 |  187 |   33 |
                                                          +------+------+----------+------+------+
                                                          49 rows in set (0.97 sec)
                                                          

                                                          We’ve had rumblings in #lobsters that there’s been too much self-promotion/content marketing recently, and there’s certainly more author-submitted posts the last four months (as @nickpsecurity observed). Maybe someone would like to write queries to get at this?

                                                          1. 3

                                                            Appreciate the data. It confirms that authored by’s went up with it being steadily over 200 since 08/2018. Our curators are pretty good at flagging away, countering or just dropping self promotion with no content. I’m not too worried about it. I probably need to get back to messaging these people about what kinds of content Lobsters prefer. I used to do it sometimes for people whose submissions got no reaction or a negative one.

                                                        1. 3

                                                          This had the useful result that you could no longer accidentally cat a directory and get all sorts of gibberish spewed on your screen, without requiring cat (and everything else that reads files) to explicitly refuse to touch directories. This feature does not seem to have spread to Solaris or the *BSDs, at least as far as I can see.

                                                          Failing a read(2) with EISDIR was introduced in OpenBSD 6.1 (released in 2017). See this comment and the rest of the thread for some additional insights into history.

                                                          1. 2

                                                            As I’m a bit ignorant about these things, just wondering if anyone can explain the motivation for the Xorg setuid change?

                                                            1. 10

                                                              A local root hole in X.org discovered around the release of 6.4: https://marc.info/?l=openbsd-tech&m=154050351216908&w=2

                                                              1. 2

                                                                https://marc.info/?l=openbsd-tech&m=154050351216908&w=2

                                                                Wow, that’s… Very interesting.

                                                                OTOH, I think it’s pretty cool that Theo would reveal such info even though it may seem to undermine credence about revealing security vulnerabilities to his own project. I mean, if his own people don’t tell him about the upcoming embargoed bugs due to the known stance of The OpenBSD Project against security embargoes, what can be expected of other projects and bigger vendors?

                                                                I’m happy to see matthieu is still a committer, though, and has been committing throughout Oct and Nov last year around this controversy, too; i.e., at least from the public eye, there’s no evidence to suggest that his account was ever disabled; but this is some harsh reality come OpenBSD way…

                                                              2. 6

                                                                Heh, I’ve used startx(1) for as long as I’ve used Unix systems, guess I have to finally start using a login manager!

                                                              1. 11

                                                                Some personal favorites:

                                                                • Work has started on a ISC-licensed rsync-compatible program called OpenRSYNC. In this release it has basic functionality such as -a, –delete, but lacks –exclude. Work will continue.
                                                                • unveil(2) has been improved to understand and find covering unveil matches above the working directory of the running process for relative path accesses. As a result many programs now can use unveil in broad ways such as unveil(”/”, “r”).
                                                                • Now using unveil(2) in ospfd(8), ospf6d(8), rebound(8), getconf(1), kvm_mkdb(8), bdftopcf(1), Xserver(1), passwd(1), spamlogd(8), spamd(8), sensorsd(8), snmpd(8), htpasswd(1), ifstated(8). Some pledge(2) changes were required to accommodate unveil.
                                                                • RETGUARD replaces the stack protector on amd64 and arm64, since RETGUARD instruments every function that returns and provides better security properties than the traditional stack protector.
                                                                • tcpdump(8) already used privsep, pledge(2) and unveil(2) containment. It now also drops root privileges completely (switching to a reserved uid).
                                                                • malloc(3) now uses sysctl(2) to get its settings, making it respect the system-wide settings in chroots as well.
                                                                • LibreSSL 2.9.1: Added support for XChaCha20 and XChaCha20-Poly1305.
                                                                • Removed some ASN.1 related code from libcrypto that had not been used since around 2000.
                                                                • ssh(1): When prompting whether to record a new host key, accept the key fingerprint as a synonym for “yes”. This allows the user to paste a fingerprint obtained out of band at the prompt and have the client do the comparison for you.
                                                                • Xorg(1), the X window server, is no longer installed setuid. xenodm(1) should be used to start X.
                                                                • Mandoc 1.14.5: Much better HTML output, in particular with respect to paragraphs, line breaks, and vertical spacing in tagged lists. Tooltips are now implemented in pure CSS, the title attribute is no longer abused.

                                                                I guess the new filtering engine of OpenSMTPD isn’t ready yet.

                                                                1. 14

                                                                  “O Deep Thought computer,” he said, “the task we have designed you to perform is this. We want you to tell us….” he paused, “The Answer.”
                                                                  “The Answer?” said Deep Thought. “The Answer to what?”
                                                                  “Life!” urged Fook.
                                                                  “The Universe!” said Lunkwill.
                                                                  “Everything!” they said in chorus.
                                                                  Deep Thought paused for a moment’s reflection.
                                                                  “Tricky,” he said finally.
                                                                  “But can you do it?”
                                                                  Again, a significant pause.
                                                                  “Yes,” said Deep Thought, “I can do it.”
                                                                  “There is an answer?” said Fook with breathless excitement.
                                                                  “Yes,” said Deep Thought. “Life, the Universe, and Everything. There is an answer. But, I’ll have to think about it.”

                                                                  Fook glanced impatiently at his watch.
                                                                  “How long?” he said.
                                                                  “Seven and a half million years,” said Deep Thought.
                                                                  Lunkwill and Fook blinked at each other.
                                                                  “Seven and a half million years…!” they cried in chorus.
                                                                  “Yes,” declaimed Deep Thought, “I said I’d have to think about it, didn’t I?”

                                                                  [Seven and a half million years later…. Fook and Lunkwill are long gone, but their descendents continue what they started]

                                                                  “We are the ones who will hear,” said Phouchg, “the answer to the great question of Life….!”
                                                                  “The Universe…!” said Loonquawl.
                                                                  “And Everything…!”
                                                                  “Shhh,” said Loonquawl with a slight gesture. “I think Deep Thought is preparing to speak!”
                                                                  There was a moment’s expectant pause while panels slowly came to life on the front of the console. Lights flashed on and off experimentally and settled down into a businesslike pattern. A soft low hum came from the communication channel.

                                                                  “Good Morning,” said Deep Thought at last.
                                                                  “Er..good morning, O Deep Thought” said Loonquawl nervously, “do you have…er, that is…”
                                                                  “An Answer for you?” interrupted Deep Thought majestically. “Yes, I have.”
                                                                  The two men shivered with expectancy. Their waiting had not been in vain.
                                                                  “There really is one?” breathed Phouchg.
                                                                  “There really is one,” confirmed Deep Thought.
                                                                  “To Everything? To the great Question of Life, the Universe and everything?”
                                                                  “Yes.”
                                                                  Both of the men had been trained for this moment, their lives had been a preparation for it, they had been selected at birth as those who would witness the answer, but even so they found themselves gasping and squirming like excited children.
                                                                  “And you’re ready to give it to us?” urged Loonsuawl.
                                                                  “I am.”
                                                                  “Now?”
                                                                  “Now,” said Deep Thought.
                                                                  They both licked their dry lips.
                                                                  “Though I don’t think,” added Deep Thought. “that you’re going to like it.”
                                                                  “Doesn’t matter!” said Phouchg. “We must know it! Now!”
                                                                  “Now?” inquired Deep Thought.
                                                                  “Yes! Now…”
                                                                  “All right,” said the computer, and settled into silence again. The two men fidgeted. The tension was unbearable.
                                                                  “You’re really not going to like it,” observed Deep Thought.
                                                                  “Tell us!”
                                                                  “All right,” said Deep Thought. “The Answer to the Great Question…”
                                                                  “Yes..!”
                                                                  “Of Life, the Universe and Everything…” said Deep Thought.
                                                                  “Yes…!”
                                                                  “Is…” said Deep Thought, and paused.
                                                                  “Yes…!”
                                                                  “Is…”
                                                                  “Yes…!!!…?”
                                                                  “the new filtering engine of OpenSMTPD isn’t ready yet,” said Deep Thought, with infinite majesty and calm.

                                                                  1. 2

                                                                    Is this the new “Is KERNSEAL ready yet?”

                                                                1. 18

                                                                  I have long had trouble understanding how the blockchain gets the very many magical properties ascribed to it. Every time I’ve asked, I have gotten a lecture in hashes and proof of work. I know the fundamentals of blockchains, what I don’t understand is how the fundamentals lead to these amazing emergent properties.

                                                                  This article sorta kinda makes me think I might not be missing anything at all – the people talking about it may have been full of shit.

                                                                  1. 8

                                                                    I have been wrong before but to me it seems like a mass psychological phenomenon. That many people and that much money cannot be wrong! So they add more people and money.

                                                                    In the best case, some companies use the label “block chain” to market some established cryptographic techniques that are not block chain at all…

                                                                    1. 7

                                                                      yep. You might think “blockchain” meant something like “append-only ledger with a consensus mechanism”, but it turns out in practice to literally just mean “whatever I’m trying to sell you today”.

                                                                      I was talking about this a few months ago with a well-meaning non-techie, who suggested that Uber - the money-burning minicab firm with an app - was an example of a “decentralised system.” More than that - it was a model for how blockchain could succeed.

                                                                      I think they’d never thought about the concept of distributed systems of any sort ever before in their lives.

                                                                      “It’s like blockchain, because anyone can sign up to be an Uber driver!”
                                                                      “Uh … anyone can sign up to be a minicab driver.”

                                                                      or the very concept of “open source” only being possible with “blockchain”.

                                                                      1. 4

                                                                        The weird thing is that I know intelligent, technical people that advocate for this. If asked for specifics, some variant of “we still have to figure out the specifics” is used.

                                                                        Well, chances are that you never will…

                                                                      2. 4

                                                                        The hype cycle became self fulfilling. I got a look at the internal roadmap for one of the pieces of legacy software at the big enterprise I work at - crusty old barely touched 90s technology that’s critical for parts management and ordering.

                                                                        2020 plans? Traceability of parts on the blockchain.

                                                                        1. 1

                                                                          Traceability, correct me if I’m wrong, was one of the actual things a distributed append-only ledger was good at. The way I see it, it’s a good decision with regards to what tech to use, at least until someone puts the wrong data in.

                                                                      3. 4

                                                                        As well as that non-tech explanation talk, I have the longer and more techy version to an infosec group. (My mission to get across to them: “please don’t get into blockchains”)

                                                                        1. 4

                                                                          You’re sending the wrong message. Instead, tell them to come up with something useful, pitch a blockchain version, build the useful thing first with the money, open source it, and then build the “better” version with blockchain. We’ll steadily get more useful stuff out of blockchain investments.

                                                                        2. 4

                                                                          I’d like to refer to this article about Bitcoin from 2011, before all the mass hysteria began: https://paulbohm.com/articles/bitcoins-value-is-decentralization/

                                                                          To elaborate: Bitcoin isn’t just a currency but an elegant universal solution to the Byzantine Generals’ Problem, one of the core problems of reaching consensus in Distributed Systems. Until recently it was thought to not be practically solvable at all, much less on a global scale. Irrespective of its currency aspects, many experts believe Bitcoin is brilliant in that it technically made possible what was previously thought impossible.

                                                                          /edit quote

                                                                          1. 4

                                                                            Herd behavior. It’s usually irrational except for the scheming people fueling and benefiting from it.

                                                                            Blockchain looks like herd behavior. Similarly, most of it has a tiny group of people that will get rich if enough buy in. That’s similar to how companies like Goldman create, pop, and profit from bubbles in the market.

                                                                            1. 3

                                                                              irrational exuberance meets unjustifi-ed/able faith in technology.

                                                                              if you bought into a blockchain, you want to hype it up because that’s how you get paid. If you didn’t buy into it, well you got bored of trying to reason with people a long time ago.

                                                                              It’s probably the most interesting social phenomenon of recent years.

                                                                              1. 2

                                                                                It sounds basically like a pyramid scheme when you put it like that…

                                                                              2. 2

                                                                                The flip side of this is that some companies are actually trying to look for the blockchain-based “killer app”, if such an app ever exists. I did develop a few blockchain based proof of concepts, which didn’t go anywhere, but there wasn’t any attempt to trick anyone. It’s just about experimenting with a new technology and see what can be done with it.

                                                                              1. 1

                                                                                I left a Git branch stall for a few weeks and, when I got back to it, it was behind by almost 60,000 commits. I think one can argue that a single person cannot possibly read all the code that is added to Windows every day, let alone read what was written during the past thirty years!

                                                                                What about peer review and quality assurance?!

                                                                                1. 28

                                                                                  It’s entirely possible that Microsoft has more than one person doing peer review and QA on Windows.

                                                                                  1. 3

                                                                                    Given that it is a monorepo, the vast majority of those commits are probably not touching the area of the codebase that you’re personally working on.

                                                                                  1. 33

                                                                                    Am I missing something, or is that a whole article? Looks like an excerpt or preface though :)

                                                                                    But from what I see, your problem isn’t a vim itself, but rather plugins and overloading yourself with them. A common mistake for Vim newcomers is that they come in and just skip the vi grammar part (calling it “the boring theory stuff”) and trying things too hard. Then they discover plugins and – more importantly – plugin managers, which lower the bar dramatically. Within such conditions, they start to feel “safe” and install as many plugins as they can handle to mimic their previous IDE, their colleagues’ IDE or any IDE they pretend to be better from, just to show off maybe.

                                                                                    And after all, you end up with a pretty much beefied up editor which you don’t really know. Okay, well, you know how to open it, type the text, save and call it a day. You know “how”, but now “why”.

                                                                                    In every editor which has been thought reasonably from the ground up (not just vi - it could be emacs, joe, sam, acme, kakoune, vis, CygnusED, EDLIN…) you need to understand the rules. It’s like speaking in new dialect of your language or swimming - won’t come overnight which sounds unpleasant in age of instant grafitication, but its long-term benefits are hard to overlook.

                                                                                    So, in other words (or as people say these days - tl;dr), your problem is that you don’t grok vi (or any other editor, as I said).

                                                                                    Of course, if you want to just click ok and go on with things, you might just spawn up your web browser^W^Wtext editor called vscode, throw a ton of plugins on yourself and cheer. Would be that faster? Maybe. But only in short-term goal or if your project isn’t going to last more than year or two (which is a standard nowadays). Or if you just don’t want to “spend time on these neckbeardy things”.

                                                                                    1. 3

                                                                                      So, in other words (or as people say these days - tl;dr), your problem is that you don’t grok vi (or any other editor, as I said).

                                                                                      I’m not sure how these words hold up to someone who calls himself a “VIM master from using others’ plugins to rolling out my own”.

                                                                                      After seeing some colleagues working at lightning speed with PhpStorm I gave up thinking vi(m) is always faster, especially on the cases the OP mentions like renaming/refactoring classes, functions and files in a big project. The reason I still use vi for everything, is because today I mainly develop in C, which is natively supported in vi and because I can use vi for everything, not only development but also sysadmin, mail composing etc.

                                                                                      1. 3

                                                                                        You said what I was thinking in more words.

                                                                                        1. 2

                                                                                          Am I missing something, or is that a whole article? Looks like an excerpt or preface though :)

                                                                                          This is ‘part 1’, so I presume they split it up to increase their exposure, which means we will probably see part 2, 3, 4,…,N of this poor article here over time.

                                                                                        1. 5

                                                                                          However, benchmarks quickly showed that wireguard-go falls very short of the performance offered by the kernel module. This is because while the Go language is very good for writing servers, it is not so good for raw packet processing, which a VPN essentially does.

                                                                                          Is the difference between Go versus Rust really that big? I would suspect the difference between kernel and userland is way bigger.

                                                                                          Rust is a modern, safe language that is both as fast as C++ and is arguably safer than Go (it is memory safe and also imposes rules that allow for safer concurrency), […].

                                                                                          Interesting point about Rust being arguably safer than Go.

                                                                                          1. 12

                                                                                            Go is pretty fast and more than enough for lots of tasks, especially in its niche of higher-level network-bound servers.

                                                                                            However, in this case there’s a bunch of cryptography, packet parsing and shuffling of data from one place to another, where LLVM optimized code, zero-cost abstractions and no-GC memory slices shine.

                                                                                          1. 3

                                                                                            Though there are already objections to do ECDH and key derivation in the kernel

                                                                                            I’ve read some objections by @zx2c4 as well about putting timers and handshake code outside of the kernel when this was proposed by someone developing a kernel version for OpenBSD: https://lists.zx2c4.com/pipermail/wireguard/2018-December/003656.html

                                                                                            In my own userland implementation I’ve separated the long term secrets into a small privileged process apart from the rest.

                                                                                            1. 2

                                                                                              In my own userland implementation I’ve separated the long term secrets into a small privileged process apart from the rest.

                                                                                              That’s the right way to do it. :)

                                                                                              EDIT: Finally reading mailing list (shorter than I thought), his counter is pretty good. We keep stuff out of the kernel by default due to higher damage upon code injection. Another principle for trusted code is keeping it simple enough to know with confidence that won’t happen. He says he did that. The last factor is whether it’s easier to analyze his code for immunity vs yours plus its interactions with kernel code. The latter might be considerably more complex. This is good example of where exception might be made. So, either sounds good in this case. One might still apply some verification or hard-hitting analyzers to it if wanting extra assurance.

                                                                                            1. 4

                                                                                              So I wasn’t sure whether or not to say I’m the author here. I’m not the author, but I am running the instance of writefreely on OpenBSD. I’m hoping to build this into a federated space for others interested in reading and writing medium-high quality content, particularly around the BSD community.

                                                                                              I’ll have an open beta from April, but if anyone’s interested in writing content here, drop me a line for an invite. It’s totally open, all your data is yours, you can read content without javascript and there’s no visitor tracking beyond post view counts accessible only by a post’s individual author.

                                                                                              It’d be really nice to get some people interested in writing about lesser known operating system experiences, Open/NetBSD use, less common FreeBSD use and so on.

                                                                                              1. 2

                                                                                                “So I wasn’t sure whether or not to say I’m the author here. I’m not the author”

                                                                                                You’re the author if you wrote the content, built the tools, etc. That’s probably the original intent. It’s also the usage in almost all situations I see “authored by.”

                                                                                                1. 3

                                                                                                  Yeah it was h3artbl33d that wrote the piece, hit publish, I just gave him an account. It’s the submit tick box I’m unsure about. I don’t want to claim authorship, but certainly there’s a connection and I’m not sure the best way to make that clear.

                                                                                                  1. 1

                                                                                                    Maybe you/chargen.one can be seen as a publisher, comparable to the publisher of a journal.

                                                                                              1. 39

                                                                                                I’m really burning out on “simplicity” posts. I get it, simplicity is good. But that doesn’t actually inform me as a developer. Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                                                                                It’s like if I stood on stage and said “Be good! Don’t be evil! Being evil is bad!” Sure, everybody agrees with that, but does it actually help people make moral choices?

                                                                                                (Also the analogy is dumb. Yes, we should totally base our engineering practice on a movie! A movie where the engineers are wrong because of magic.)

                                                                                                1. 11

                                                                                                  Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                                                                                  Because your questions are difficult and answers are dependent on a lot of factors.

                                                                                                  I’ll tell you what I do to detect simplicity, maybe you’ll find it useful. Let’s start with a real-life example.

                                                                                                  I needed tokens for authorization, I reviewed existing formats, JWTs look conservative and Macaroons look powerful.

                                                                                                  What do I do? I dissect the formats. For JWTs I read the RFCs and implemented software to create them and verify them (each in 2 languages) for various options (key algorithms).

                                                                                                  For Macaroons I read the whitepaper, then implemented verifier based on the whitepaper, reviewed existing implementations, found out differences between the whitepaper and de-facto code with explanations. While comparing my implementation I found out some security issues with existing code. Additionally I implemented the rest of the stack (de/serialization, UI for manipulation of Macaroons). After two months I knew precisely where does complexity lie in Macaroons and of course there are the only spots all blogposts don’t mention (spoilers: cycles in third party caveats, no standards for encoded caveats…)!

                                                                                                  Then I looked at my JWT proof-of-concept code - it uses base64(url) and JSON, primitives that basically all programming environments have built-in. After limiting the algorithms used the entire verifier takes just a couple of lines of code! It’s vastly simpler than the Macaroon one.

                                                                                                  What’s the moral here? That you need a lot of time to see for yourself what is simple and what is complex. Now every time I see a post recommending Macaroons I can already see the author didn’t use them in practice (compare that with the Tess Rinearson post linked at the end of that article).

                                                                                                  That’s only the example, I routinely implement various protocols and re-implement software (ActivityPub, Mailing Lists, roughtime client) and each time I discover what’s simple or what’s complex in each one of them.

                                                                                                  (By the way your book is excellent!)

                                                                                                  1. 9

                                                                                                    I get it, simplicity is good.

                                                                                                    Alas, not everybody gets it. The best that these kinds of exhortations can do (all that they aim to do, as far as I can tell) is to persuade people to modify their own set of values. This doesn’t immediately result in better code… but I think it’s a necessary precondition. The only developers who will even ask the good questions you suggest (let alone look for good answers) are the developers who hold simplicity as a value.

                                                                                                    (The analogy is pretty dumb though, and not especially motivating.)

                                                                                                    1. 10

                                                                                                      I’ve never met a developer who does not claim to hold simplicity as a value. But as a concept it is so subjective that this is meaningless. It’s extremely common for two developers arguing for opposing approaches each to claim that their approach is the simpler one.

                                                                                                      1. 7

                                                                                                        I get the value of exhortations. I think more examples would be better. Pairs of solutions where the simple one meets requirements with a number of better attributes. Developers often prefer to see the difference and benefits instead of being told.

                                                                                                      2. 6

                                                                                                        Exactly. This is one of those things you can’t explain in a book. When to compose, when to decompose. When to extract methods, when to inline methods. When to add a layer of abstraction, when to remove one. When is it too flexible, when is it too simplistic?

                                                                                                        No amount of rules of thumb is going to answer those question. I only know of one way to learn it: practice. Which takes effort and most importantly, time. Rendering this kind of posts mostly useless.

                                                                                                        1. 3

                                                                                                          P.S. They do feel good to write though, so people will keep writing them, and there’s nothing wrong with it either.

                                                                                                        2. 5

                                                                                                          I agree that anecdotes like this can get old, but I’ve been meaning to actually write a similar post to this… on something I’ve been calling the “too many buttons” syndrome. This issue pops up a ton in large pieces of software (Though I’m specifically thinking of projects like JRA and Confluence) where there’s an option for everything.

                                                                                                          Not everyone gets that simplicity is good because it can be harder to sell. “If a user wants it, we should do it” is something I’ve heard just a few too many times without bothering to look at the use case or if it could be done better. Sometimes it’s worth stepping back and looking at the complexity something will add to the project (in both code and testing… especially when it comes to options and how they interact with each other) rather than just adding all the little features.

                                                                                                          1. 5

                                                                                                            In my experience a lot of commercial companies that develop under tight deadlines produce a lot of suboptimal and dreadful code. Often it takes more time, to produce less code simply because the more time you spend on a difficult problem, the better you understand it. I think the reason that most a lot of software is bloated and complex is because it’s “good enough” which is optimal from an economic point of view.

                                                                                                            The other day there was a discussion here on Lobsters about all the required pieces needed to run a Mastodon instance and the popular solution of abstracting all that away in a Docker container. There are alternative implementations that depend on a smaller number of components alleviating the need for dumping everything in a container (of course the question is, do these alternatives offer the same functionality).

                                                                                                            How do we detect simplicity?

                                                                                                            For me personally simplicity has to do with readability, maintainability and elegance of code or infrastructure. If someones solution involves three steps, and someone else can do it in two steps (with comparable cognitive load per step), I would say it’s more simple.

                                                                                                            How do we know when we shouldn’t simplify?

                                                                                                            If that would cut some features you cannot miss.

                                                                                                            1. 5

                                                                                                              You are so right. After years of experience, I only start to clarify my idea of “simplicity”. There are different kind of simplicity most of them are not totally compatible. And in my opinion some need to be preferred to other, but there is no clear rule. To make a choice between different complexity I still use a lot of intuition and I debate a lot, and I am still unsure my choice are the best.

                                                                                                              • only using basic feature of a language (do not use advanced programming language feature) is certainly the most important aspect in simplicity. It will make your code easy to read by more people.
                                                                                                              • don’t use too much intermediate functions, and if possible don’t disperse those function in many different files before really feel you are copy/pasting too much. My rule of thumb is, 2 or 3 times duplications is totally fine and superior to centralisation of code. It start to be really clear that code factorisation is good when you start repeating yourself more than 6 to 10 times
                                                                                                              • only really use advanced feature of the language after having tried not to use it for some time and really lack the ability of that advanced feature. Some examples of what I call advanced feature of a language are; class heritage, protocols in Clojure, writing your own typeclasses in Haskell, meta programming (macros in LISP), etc…
                                                                                                              • prefer stateless functions to objects/service with internal states
                                                                                                              • prefer pure functions (side effect free) other procedures (functions with side effects)
                                                                                                              • give a lot of preference to composable solutions ; composable in the algebraic meaning. For example, I do my best not to use LISP macros, because most of the time macros break composability. The same could be said when you start to deal with type-level programming in Haskell, or when you are doing meta-programming in ruby/python.

                                                                                                              For now, all those rules are still quite artisanal. I don’t have any really hard metrics or strong rules. Everything I just said is “preferable” but I’m pretty sure we can find exception to most of those rules.

                                                                                                              1. 5

                                                                                                                Amen, +1, etc. “Simplicity” often just means that a concept fits cleanly in the maker’s head at a particular point in time. How many times have I returned to a project I thought was simple only to find I had burdened it with spooky magic because I didn’t benefit from critical distance at the time? When was the last time I deemed another person’s work “too complex” because I couldn’t understand it in one sitting and wasn’t aware of the constraints they were operating under? Answers: too often and too recently.

                                                                                                                1. 3

                                                                                                                  What kinds of simplicity are there?

                                                                                                                  This is a good question (as are the others). Borrowing from Holmes, I’d say there’s a continuum from naive simplicity, to complexity, to simplicity on the other side of complexity (which is what is truly interesting)

                                                                                                                  For example, “naively simple” code would only cover a small subset (say, the happy path) of a business problem. Complex code would handle all, or most, of the business complexity but in a messy, complicated way. “Other side” simplicity refines that complex code into something that can handle the business complexity without itself becoming overly complicated.

                                                                                                                  1. 2

                                                                                                                    What happens to simplicity? We trade it for a other things of course. For example, you can have simple regular expressions, but most people prefer less simple and more powerful implementation like Perls.

                                                                                                                    Simplicity is often a tradeoff versus easyness, performance, flexibility, reusability, useability, etc. So simplicity is good, but those other things are also good.

                                                                                                                    1. 1

                                                                                                                      Most people seem to agree that simplicity is best. However, when it comes down to simplicity for the user versus the developer, I have seen disagreement. Each trade off is going to be situation and implementation dependent, but at my job I’ve been pushing for a simpler developer environment.

                                                                                                                      In my office, there is a tendency to create exceptions to rules because it makes things simpler for the user. Since the environment has more exceptional circumstances, it tends to have more errors when people forget the undocumented exception case. In my opinion, this causes an uneven experience for the user despite being “simpler.”

                                                                                                                      My experience is coming from a medium sized, non-tech company. I work in the IT department so we are a cost center. There is an emphasis on white glove treatment of the revenue producing portions of the company. YMMV