1. 25

    That’s a good counter argument to “why pull in dependencies if I can write my own in 20 lines of code”

    1. 2

      I mean, its more like:

      if your program is a bit slow, spend a few minutes to understand why.

      I dunno if it has anything to do with dependencies or not.

      1. 6

        There are JSON parsers that parse at 400-800MB/s. There are SIMD-optimized hashmaps.

        Get these two, and the 60MB file would have been processed in a fraction of a second. You wouldn’t even need to dig out a profiler and reinvent these two very well reinvented wheels.

        I’m not saying you shouldn’t profile your code, but choose your battles. Profiling effort could be spent on optimizing something unique to the game, rather than solving an already solved problem.

    1. 6

      Remember using upx could be a horrible idea for programs that have more than once instance running at once. Normally the kernel can share memory pages between different executions of the same program, upx defeats that.

      1. 2

        upx also doesn’t fix that the binary is (likely) not very cache-efficient

        1. 1

          I didn’t know about that specifically, so I’ll add a note about it. I kind of discounted dynamic linking as soon as I realized modern cloud deployment practices would mean only one executable per instance/VM, and I didn’t consider much by way of sharing instances after that

          Thanks for the feedback!

          1. 2

            On that topic, I wonder if you can reproduce the gains of UPX with some standard (non-autoextracting) compression. If that the case, then you might also gain the same benefits from on-the-wire compression when transferring the executable.

            1. 1

              Theoretically, the OS’ executable format can include compressing the pages, so then the kernel can expand the pages in a way that’d be shared across all packages.

        1. 4

          This got me tempted to build Janet this way. I’ve got 127 lines of errors to go through but I hope I can get it done (the resulting binary should be small too).

          It would be nice if the author could share his build commands and stubs. Might help some other efforts :)

          1. 3

            My advice - enable reduced OS mode, disable threads, disable ev in the config.h .

            1. 3

              The author forked the lua repo to be cosmopolitan compatible. Commit is available at

              https://github.com/ahgamut/lua/commit/aa20d02dffc4f96d673d30e7378c0c02a6df13f8

            1. 3

              Lite’s implementation is one of the most beautiful I’ve seen. It’s one of those programs that you read and it’s just…elegant and beautiful.

              1. 3

                I took a look at both the Lua and C code, and it looks nice… although I prefer code with more comments. There are many files without a single comment. Even 1 or 2 well-chosen sentences at the top of the file goes a long way.

                https://github.com/rxi/lite/tree/master/data/core

                1. 2

                  I mean, this whole overview kind of makes what you are asking for redundant.

                  1. 1

                    I’m often told that I don’t comment my code enough, so maybe that’s why I like Lite so much. :)

                1. 2

                  I wish rust had just added a ‘emit portable C’ option from day one.

                  1. 4

                    That was never the goal. However if you want, then there is alternative compiler mrustc that supports that.

                  1. 8

                    For the example, its actually really useful to allow a program to listen on either a unix socket or a tcp socket just by a command line flag. The string arg is useful when passed through to users.

                    1. 9

                      Yes. The fact that an address is a mostly-opaque string to programs allowed plan 9 to add support for ipv6, without touching userspace networking APIs at all.

                      Often, the best thing your program can do is treat data the user passes it as an opaque blob, for as much of its travel through your code as possible.

                      1. 3

                        Yeah I would call that a form of polymorphism, just like file descriptors.

                        Unix could have had different data types for disk files, pipes, and sockets, instead of using integers, but then you would lose the polymorphism. You wouldn’t be able to write select(), and shell couldn’t have redirects.

                        It also same caveats because there are some things you can do on disk files that you can’t do on the others (seek()).

                        Still, given the constraints of C, the untyped design is better.

                        This relates to my comment about textual protocols in Unix: https://lobste.rs/s/vl9o4z/case_against_text_protocols#c_wsdhsm (which will hopefully appear on my blog in the near future)

                        Basically text is “untyped” and therefore you get generic operations / polymorphism (copy, diff, merge, etc.).

                        Types can inhibit composition.


                        (I know somebody is going to bring up more advanced type systems. I still want to see someone write a useful operating system that way. I’m sure it can be done but it hasn’t so far AFAIK. Probably somewhat because of the expression problem – i.e. you want extensibility in both data types (files, sockets) and operations (read, write, seek) across a stable API boundary. )

                        1. 1

                          Basically text is “untyped” and therefore you get generic operations / polymorphism (copy, diff, merge, etc.).

                          “Sir, this is an Arby’s.”

                          (Go already has this problem. I do not see your point, nor your point about the expression problem, as the number of address types is enumerable, and set in stone..)

                          1. 2

                            The “challenge” is to show me an operating system WITH fine-grained types that does NOT have the O(M*N) explosion of code. That is, show me how it solves the polymorphism problem. Here’s a canonical situation of M data types and N operations, and you can generalize it with more in each dimension (M and N both get bigger):

                            • The data types: the operating system has persistent disk files, IPC like pipes, and networking to remote machines.
                            • Now for the “operations”
                              • How do you simultaneously wait on events from files, IPC and networks? Like select() or inotify().
                                • Ditto with signals and process exits – waitfd(), signalfd(), etc. (Unix fails here so Linux invented a few more mechanisms).
                              • How do you do redirects? A shell can redirect from a file or a pipe. (Both Rob Pike and DJB complained about the lack of compositionality for sockets: https://cr.yp.to/tcpip/twofd.html. Pike doesn’t like the Berkeley socket API because it’s non-compositional. That’s why both Plan 9 and Go have something different AFAIK)
                              • How do you copy from a disk to IPC, disk to network, IPC to network, etc. In Unix, cat can work in place of cp. netcat also sort of works, but I think Plan 9 does better because networking is more unified.

                            The connection is:

                            • When you have fine-grained types, you get O(M * N) problems, and the expression problem arises out of that. You have M data types and N operations.
                            • If you have a single type like a file descriptor or a string, then you don’t have an O(M * N) problem. So you don’t have a composition problem, and the resulting explosion of code.

                            In other words, types can inhibit composition. Again, not saying it can’t be done, but just that I haven’t seen an OS that addresses this.

                            Plan 9 is more compositional precisely because it has FEWER types, not more. As mentioned, opaque strings are used as addresses.

                            (Rich Hickey also has a great talk on this about the Java HTTP Request interface. It’s a type in the Java standard library that inhibits composition and generic operations. Link appreciated from anyone who knows what I’m talking about.)


                            Kubernetes has a similar issue as far as I can tell:

                            https://twitter.com/n3wscott/status/1355550715519885314

                            I’m not familiar with the details, but I used the “predecessor” Borg for many years and it definitely has some composition problems. The Kubenetes ecosystem has a severe O(M * N) code explosion problem. Unix has much less of that, and Plan 9 probably has even less.

                            Widely mocked diagram: https://twitter.com/QuinnyPig/status/1328689009275535360

                            Claim: This is an O(M * N) code explosion due to lack of compositionality in Kubernetes.


                            Also read Ken Thompson’s “sermonette” in his paper on the design of Unix shell:

                            https://lobste.rs/s/asr9ud/unix_command_language_ken_thompson_1976#c_1phbzz

                            A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion.

                            “Single notion” means ONE TYPE. Now this is taken to an extreme – Unix obviously does have both file descriptors and strings. But it uses those concepts for lots of things that would be modelled as separate types in more naive systems.


                            Many familiar computing ‘concepts’ are missing from UNIX. Files have no records. There are no access methods. User programs contain no system buffers. There are no file types. These concepts fill a much-needed gap.

                            Records are types. Unix lacks records. That’s a feature and not a bug. Records should be and ARE layered on top.

                            In distributed systems, types should be layered on top of untyped byte streams.

                            (This is going to be on the Oil blog. Many people have problems seeing this because it’s an issue of architecture and not code. It’s a systems design issue.)

                            1. 1

                              The problem is that different instances support different functionality. Seek works on some file descriptors, but not others. Ioctl is a horrific mess.

                              Even COM was a better solution.

                        2. 2

                          without touching userspace networking APIs at all.

                          This kind of statement sounds amazing, but has about 10 *’s next to it pointing out all the caveats. User space code still needed to change to account for new address inputs, otherwise who’s to say that “window.alert()” isn’t a valid address? You’re gonna pass arbitrary strings to that syscall?

                          No. Create a constraint on the interface that ensures the caller isn’t an idiot.

                          1. 4

                            who’s to say that “window.alert()” isn’t a valid address?

                            “window.alert()” IS a valid address for a unix domain socket. Maybe other things too.

                            1. 1

                              Unfortunate random string choice. :)

                            2. 3

                              How much risk does that actually mitigate in practice, and how much toil does it create?

                              It is not true that stronger type systems, or stronger constraints, are strictly better in all circumstances.

                              1. 5

                                and how much toil does it create?

                                In the example, the senior programmer spent nearly a day on this. It could have been a type error.

                                It is not true that stronger type systems, or stronger constraints, are strictly better in all circumstances.

                                I am not a static types apologist, and prefer to write stuff in Lispy languages, which are often quite dynamic, but I will never concede that two parameters that have a dependency on each other, as in a pair such as (socket type, address), are best represented by two arbitrary strings instead of an enumeration carrying a constrained value. That’s an absurd thing to argue. You’ve created an infinitely large set of potential inputs and told the programmer “Don’t worry! We’ll tell you if you’re wrong when you run the program.” How silly, especially when that state space can be reduced significantly, and the usage checked by the compiler we’re already paying for.

                                1. 2

                                  In the example, the senior programmer spent nearly a day on this. It could have been a type error.

                                  Yep, that’s definitely toil, and it definitely could have been prevented by the compiler. But what about the toil that those type system features would bring to every other aspect of programming in the language? How do you measure that, and how do you weigh it? And not in the abstract, either, but in the specific context of Go as it exists today. Honest questions. I don’t know! But I do know it’s important to do, if you want a true reckoning of the calculus.

                                  Of course, this could also have been caught and prevented by a unit test. Or an integration test. Or during code review. Or by using an architectural model that was a better fit to the problem. There are many tools available to software engineering teams to mitigate risks, and each tool carries its own costs and benefits as it is applied in different contexts. If the game is mitigating risk — which it is — then not everything needs to be solved at the language level.

                                  I will never concede that two parameters that have a dependency on each other, as in a pair such as (socket type, address), are best represented by two arbitrary strings instead of an enumeration carrying a constrained value.

                                  Sure, that sounds good to me, but it’s much more abstract than the specific claim made in the article, which is that

                                  net.Dial("tcp", "1.2.3.4:9090")
                                  

                                  should become

                                  net.Dial(net.TCPIPAddr{
                                      IP:   net.IPv4(1, 2, 3, 4),
                                      Port: 9090,
                                  })
                                  

                                  This would certainly convert a class of bugs that are currently runtime errors into compile time errors. But it would also make every use of net.Dial substantially more laborious. Is the benefit worth the cost? I don’t think the answer is obviously yes or no.

                                2. 3

                                  I don’t think it’s possible to have a clear answer to this—but FWIW, my experience is that in ecosystems where unioned string literal types are deployed widely (notably in TypeScript), there is effectively no meaningful dissent about whether that solution is strictly better than naked-string-only typing, in every practical measurable dimension. It makes docs better, it makes tooling and autocomplete better, it seems to prevent bugs, people generally just seem to like using it, and there are no appreciable downsides in practice (compile time impact is negligible, no implications for generics, etc.).

                                  I understand that in Go this does not quite pass muster, because Go’s methodology for evaluating language features implicitly (and significantly) discounts what most other languages would call “ergonomic” improvements. Other ecosystems are willing to rely on intuition for deciding what sorts of day-to-day activities are worth improving, and in those ecosystems, it is harder to argue that we should not make better type amenities around the most important data type in the field of CS (i.e., the string) for things that people do all the time (e.g., pass in “stringly-typed” arguments to functions), especially when there are no significant downsides (e.g., compilation speed, design subtlety, etc.).

                                3. 3

                                  You’re gonna pass arbitrary strings to that syscall?

                                  To the (userspace) connection server, which decides how to best reach the address specified, and returns an appropriate error if the caller provides garbage – yes. Why not? Having one place that does all the work shared across the system makes it easy to have a single, consistent, interface with one location to search for bugs.

                                  There are, of course, a few asterisks: for example, the dns resolver needs to be taught about how to handle AAAA records, 6in4 tunnels need to know how to encapsulate, etc – but the programs that need this knowlege are generally the programs that provide the userspace APIs.

                                  No. Create a constraint on the interface that ensures the caller isn’t an idiot.

                                  Opacity is the strictest possible constraint on an interface. You may do nothing with the data, other than pass it on to something else. If the caller may not do anything, then the caller will not do anything stupid.

                                  1. 1

                                    Opacity is the strictest possible constraint on an interface. You may do nothing with the data, other than pass it on to something else. If the caller may not do anything, then the caller will not do anything stupid.

                                    Ok? But it’s not at all opaque in these examples. The two parameters actually relate to each other….

                                    1. 1

                                      Yes. That’s a poor choice in the Go API – it should have been a single opaque blob, instead of two dependent ones. Plan 9 does not make this mistake.

                            1. 5

                              Betteridge’s law strikes again.

                              One of the key features of a blockchain, which the author tries to handwave away, is that every link in the chain is verifiable, and unalterable. The author tries to claim that because each commit carries a reference to its parent, it’s a “chain of blocks”, but it’s not so much a chain as just an order. You can edit the history of a git repo easily, reparent, delete, squash, and perform many other operations that entirely modify the entire chain. It was kinda made that way.

                              1. 12

                                The technical properties of git’s and common block chain data structures are relatively similar.

                                You can also fork a bitcoin block chain and pretend that your fork is the canonical one. The special bit about block chains is that there’s some mechanism for building agreement about the HEAD pointer. Among other things, there’s no designated mover of that pointer (as in a maintainer in a git-using project), but an algorithm that decides which among competing proposals to take.

                                1. 16

                                  They are technically similar because both a blockchain and a git repo are examples of a merkle tree. As you point out though the real difference is in the consensus mechanism. Git’s consensus mechanism is purely social and mostly manual. Bitcoin’s consensus mechanism is proof of work and mostly automated.

                                  1. 2

                                    Please stop referring to “Proof of _” as a consensus mechanism. It is an anti-sybil mechanism, the consensus mechanism is called “longest chain” or “nakomoto consensus” - you can use a different anti-sybil mechanism with the same consensus mechanism (but you may lose some of the properties of bitcoin).

                                    The point is that there are various different combinations available of these two components and conflating them detracts from people’s ability to understand what is going on.

                                    1. 2

                                      You are right. I was mixing definitions there. Thanks for pointing it out. The main point still stands though. The primary distinction between a blockchain and git is the consensus mechanism and not the underlying merkle tree datastructure that they both share.

                                    2. 1

                                      Mandatory blockchain != bitcoin. Key industrial efforts listed in https://wiki.hyperledger.org/ are mostly not proof-of-work in any way (the proper term for this is permissioned blockchain, which is where industrial applications are going).

                                      1. 2

                                        You are correct. I don’t disagree at all. I used bitcoin as an example because it’s well known. There are lots of different blockchains with different types of consensus mechanisms.

                                  2. 2

                                    You can make a new history but it will always be distinct from the original one.

                                    I think what you’re really after is the fact that there is no one to witness that things like the author and the date of a commit are genuine – that is, it’s not just that I can edit the history, I can forge a history.

                                    1. 1

                                      what was all that hullabaloo about git moving away from SHA-1 due to vulnerabilities? why where they using a cryptographic hash function in the first place?

                                      what you said makes sense, but it seems to suggest this SHA-1 thing was a bit of bikeshedding or theater

                                      1. 2

                                        Git uses a cryptographic hash function because it wants to be able to assume that collisions never occur, and the cost of doing so isn’t too large. A collision was demonstrated in SHA-1 in 2017.

                                        1. 3

                                          SHA-1 still prevents accidental collisions. Was Git really designed to be robust against bad actors?

                                          1. 2

                                            ¯_(ツ)_/¯

                                            1. 1

                                              The problem is that it was never properly defined what properties people expect from Git.

                                              You can find pieces of the official Git documentation and public claims by Linus Torvalds that are seemingly in contradiction to each other. And the whole pgp signing part does not seem to be very well thought through.

                                          2. 2

                                            Because you can sign git commits and hash collisions ruins that.

                                            1. 1

                                              ah that makes some sense

                                          3. 1

                                            Technically you haven’t really made the others disappear. They are all still there just not easily viewed without using reflog. All you are really doing is creating a new branch point and moving the branch pointer to the head of that new branch when you do those operations. But to the average user it appears that you have edited history.

                                          1. 37

                                            Hello, I am here to derail the Rust discussion before it gets started. The culprit behind sudo’s vast repertoire of vulnerabilities, and more broadly of bugs in general, is accountable almost entirely to one matter: its runaway complexity.

                                            We have another tool which does something very similar to sudo which we can compare with: doas. The portable version clocks in at about 500 lines of code, its man pages are a combined 157 lines long, and it has had two CVEs (only one of which Rust would have prevented), or approximately one every 30 months.

                                            sudo is about 120,000 lines of code (100x more), its had 140 CVEs, or about one every 2 months since the CVE database came into being 21 years ago. Its man pages are about 10,000 lines and include the following:

                                            $ man sudoers | grep -C1 despair
                                            The sudoers file grammar will be described below in Extended Backus-Naur
                                            Form (EBNF).  Don't despair if you are unfamiliar with EBNF; it is fairly
                                            simple, and the definitions below are annotated.
                                            

                                            If you want programs to be more secure, stable, and reliable, the key metric to address is complexity. Rewriting it in Rust is not the main concern.

                                            1. 45

                                              its had 140 CVEs

                                              Did you even look at that list? Most of those are not sudo vulnerabilities but issues in sudo configurations distros ship with. The actual list is more like 39, and a number of them are “disputed” and most are low-impact. I didn’t do a full detailed analysis of the issues, but the implication that it’s had “140 security problems” is simply false.

                                              sudo is about 120,000 lines of code

                                              More like 60k if you exclude the regress (tests) and lib directories, and 15k if you exclude the plugins (although the sudoers plugin is 40k lines, which most people use). Either way, it’s at least half of 120k.

                                              Its man pages are about 10,000 lines and include the following:

                                              12k, but this also includes various technical documentation (like the plugin API); the main documentation in sudoers(1) is 741 lines, and sudoers(5) is 3,255 lines. Well under half of 10,000.

                                              We have another tool which does something very similar to sudo which we can compare with: doas.

                                              Except that it only has 10% of the features, or less. This is good if you don’t use them, and bad if you do. But I already commented on this at HN so no need to repeat that here.

                                              1. 12

                                                You’re right about these numbers being a back-of-the-napkin analysis. But even your more detailed analysis shows that the situation is much graver with sudo. I am going to include plugins, becuase if they ship, they’re a liability. And their docs, because they felt the need to write them. You can’t just shove the complexity you don’t use and/or like under the rug. Heartbleed brought the internet to its knees because of a vulnerability in a feature no one uses.

                                                And yes, doas has 10% of the features by count - but it has 99% of the features by utility. If you need something in the 1%, what right do you have to shove it into my system? Go make your own tool! Your little feature which is incredibly useful to you is incredibly non-useful to everyone else, which means fewer eyes on it, and it’s a security liability to 99% of systems as such. Not every feature idea is meritous. Scope management is important.

                                                1. 9

                                                  it has 99% of the features by utility

                                                  Citation needed.

                                                  what right do you have to shove it into my system?

                                                  Nobody is shoving anything into your system. The sudo maintainers have the right to decide to include features, and they’ve been exercising that right. You have the right to skip sudo and write your own - and you’ve been exercising that right too.

                                                  Go make your own tool!

                                                  You’re asking people to undergo the burden of forking or re-writing all of the common functionality of an existing tool just so they can add their one feature. This imposes a great cost on them. Meanwhile, including that code or feature into an existing tool imposes only a small (or much smaller) cost, if done correctly - the incremental cost of adding a new feature to an existing system.

                                                  The key phrase here is “if done correctly”. The consensus seems to be that sudo is suffering from poor engineering practices - few or no tests, including with the patch that (ostensibly) fixes this bug. If your software engineering practices are bad, then simpler programs will have fewer bugs only because there’s less code to have bugs in. This is not a virtue. Large, complex programs can be built to be (relatively) safe by employing tests, memory checkers, good design practices, good architecture (which also reduces accidental complexity) code reviews, and technologies that help mitigate errors (whether that be a memory-safe GC-less language like Rust or a memory-safe GC’ed language like Python). Most features can (and should) be partitioned off from the rest of the design, either through compile-time flags or runtime architecture, which prevents them from incurring security or performance penalties.

                                                  Software is meant to serve the needs of users. Users have varied use-cases. Distinct use-cases require more code to implement, and thereby incur complexity (although, depending on how good of an engineer one is, additional accidental complexity above the base essential complexity may be added). If you want to serve the majority of your users, you must incur some complexity. If you want to still serve them, then start by removing the accidental complexity. If you want to remove the essential complexity, then you are no longer serving your users.

                                                  The sudo project is probably designed to serve the needs of the vast majority of the Linux user-base, and it succeeds at that, for the most part. doas very intentionally does not serve the needs of the vast majority of the linux user-base. Don’t condemn a project for trying to serve more users than you are.

                                                  Not every feature idea is meritous.

                                                  Serving users is meritous - or do you disagree?

                                                  1. 6

                                                    Heartbleed brought the internet to its knees because of a vulnerability in a feature no one uses.

                                                    Yes, but the difference is that these are features people actually use, which wasn’t the case with Heartleed. Like I mentioned, I think doas is great – I’ve been using it for years and never really used (or liked) sudo because I felt it was far too complex for my needs, before doas I just used su. But I can’t deny that for a lot of other people (mainly organisations, which is the biggest use-case for sudo in the first place) these features are actually useful.

                                                    Go make your own tool! Your little feature which is incredibly useful to you is incredibly non-useful to everyone else

                                                    A lot of these things aren’t “little” features, and many interact with other features. What if I want doas + 3 flags from sudo + LDAP + auditing? There are many combinations possible, and writing a separate tool for every one of them isn’t really realistic, and all of this also required maintenance and reliable consistent long-term maintainers are kind of rare.

                                                    Scope management is important.

                                                    Yes, I’m usually pretty explicit about which use cases I want to solve and which I don’t want to solve. But “solving all the use cases” is also a valid scope. Is this a trade-off? Sure. But everything here is.

                                                    The real problem isn’t so much sudo; but rather that sudo is the de-facto default in almost all Linux distros (often installed by default, too). Ideally, the default should be the simplest tool which solves most of the common use cases (i.e. doas), and people with more complex use cases can install sudo if they need it. I don’t know why there aren’t more distros using doas by default (probably just inertia?)

                                                    1. 0

                                                      What if I want doas + 3 flags from sudo + LDAP + auditing?

                                                      Tough shit? I want a pony, and a tuba, and barbie doll…

                                                      But “solving all the use cases” is also a valid scope.

                                                      My entire thesis is that it’s not a valid scope. This fallacy leads to severe and present problems like the one we’re discussing today. You’re begging the question here.

                                                      1. 3

                                                        Tough shit? I want a pony, and a tuba, and barbie doll…

                                                        This is an extremely user-hostile attitude to have (and don’t try claiming that telling users with not-even-very-obscure use-cases to write their own tools isn’t user-hostile).

                                                        I’ve noticed that some programmers are engineers that try to build tools to solve problems for users, and some are artists that build programs that are beautiful or clever, or just because they can. You appear to be one of the latter, with your goal being crafting simple, beautiful systems. This is fine. However, this is not the mindset that allows you to build either successful systems (in a marketshare sense) or ones that are useful for many people other than yourself, for previously-discussed reasons. The sudo maintainers are trying to build software for people to use. Sure, there’s more than one way to do that (integration vs composition), but there are ways to do both poorly, and claiming the moral high ground for choosing simplicity (composition) is not only poor form but also kind of bad optics when you haven’t even begun to demonstrate that it’s a better design strategy.

                                                        My entire thesis is that it’s not a valid scope.

                                                        A thesis which you have not adequately defended. Your statements have amounted to “This bug is due to sudo’s complexity which is driven by the target scope/number of features that it has”, while both failing to provide any substantial evidence that this is the case (e.g. showing that sudo’s bugs are due to feature-driven essential complexity alone, and not use of a memory-unsafe language, poor software engineering practices (which could lead to either accidental complexity or directly to bugs themselves), or simple chance/statistics) and not actually providing any defense for the thesis as stated. Assume that @arp242 didn’t mean “all” the usecases, but instead “the vast majority” of them - say, enough that it works for 99.9% of users. Why is this “invalid”, exactly? It’s easy for me to imagine the argument being “this is a bad idea”, but I can’t imagine why you would think that it’s logically incoherent.

                                                        Finally, you have repeatedly conflated “complexity” and “features”. Your entire argument is, again, invalid if you can’t show that sudo’s complexity is purely (or even mostly) essential complexity, as opposed to accidental complexity coming from being careless etc.

                                                  2. 9

                                                    I dont’t think “users (distros) make a lot of configuration mistakes” is a good defence when arguing if complexity is the issue.

                                                    But I do agree about feature set. And I feel like arguing against complexity for safety is wrong (like ddevault was doing), because systems inevitably grow complex. We should still be able to build safe, complex systems. (Hence why I’m a proponent of language innovation and ditching C.)

                                                    1. 11

                                                      I dont’t think “users (distros) make a lot of configuration mistakes” is a good defence when arguing if complexity is the issue.

                                                      It’s silly stuff like (ALL : ALL) NOPASSWD: ALL. “Can run sudo without a password” seems like a common theme: some shell injection is found in the web UI and because the config is really naïve (which is definitely not the sudo default) it’s escalated to root.

                                                      Others aren’t directly related to sudo configuration as such; for example this one has a Perl script which is run with sudo that can be exploited to run arbitrary shell commands. This is also a common theme: some script is run with sudo, but the script has some vulnerability and is now escalated to root as it’s run with sudo.

                                                      I didn’t check all of the issues, but almost all that I checked are one of the above; I don’t really see any where the vulnerability is caused directly by the complexity of sudo or its configuration; it’s just that running anything as root is tricky: setuid returns 432 results, three times that of sudo, and I don’t think that anyone can argue that setuid is complex or that setuid implementations have been riddled with security bugs.

                                                      Other just mention sudo in passing by the way; this one is really about an unrelated remote exec vulnerability, and just mentions “If QCMAP_CLI can be run via sudo or setuid, this also allows elevating privileges to root”. And this one isn’t even about sudo at all, but about a “sudo mode” plugin for TYPO3, presumably to allow TYPO3 users some admin capabilities without giving away the admin password. And who knows why this one is even returned in a search for “sudo” as it’s not mentioned anywhere.

                                                      1. 3

                                                        it’s just that running anything as root is tricky: setuid returns 432 results, three times that of sudo

                                                        This is comparing apples to oranges. setuid affects many programs, so obviously it would have more results than a single program would. If you’re going to attack my numbers than at least run the same logic over your own.

                                                        1. 2

                                                          It is comparing apples to apples, because many of the CVEs are about other program’s improper sudo usage, similar to improper/insecure setuid usage.

                                                          1. 2

                                                            Well, whatever we’re comparing, it’s not making much sense.

                                                            1. If sudo is hard to use and that leads to security problems through its misusage, that’s sudo’s fault. Or do you think that the footguns in C are not C’s fault, either? I thought you liked Rust for that very reason. For this reason the original CVE count stands.
                                                            2. But fine, let’s move on on the presumption that the original CVE count is not appropriate to use here, and instead reference your list of 39 Ubuntu vulnerabilities. 39 > 2, Q.E.D. At this point we are comparing programs to programs.
                                                            3. You now want to compare this with 432 setuid results. You are comparing programs with APIs. Apples to oranges.

                                                            But, if you’re trying to bring this back and compare it with my 140 CVE number, it’s still pretty damning for sudo. setuid is an essential and basic feature of Unix, which cannot be made any smaller than it already is without sacrificing its essential nature. It’s required for thousands of programs to carry out their basic premise, including both sudo and doas! sudo, on the other hand, can be made much simpler and still address its most common use-cases, as demonstrated by doas’s evident utility. It also has a much smaller exposure: one non-standard tool written in the 80’s and shunted along the timeline of Unix history every since, compared to a standardized Unix feature introduced by DMR himself in the early 70’s. And setuid somehow has only 4x the number of footgun incidents? sudo could do a hell of a lot better, and it can do so by trimming the fat - a lot of it.

                                                            1. 3

                                                              If sudo is hard to use and that leads to security problems through its misusage, that’s sudo’s fault.

                                                              It’s not because it’s hard to use, it’s just that its usage can escalate other more (relatively) benign security problems, just like setuid can. This is my point, as a reply to stephank’s comment. This is inherent to running anything as root, with setuid, sudo, or doas, and why we have capabilities on Linux now. I bet that if doas would be the default instead of sudo we’d have a bunch of CVEs about improper doas usage now, because people do stupid things like allowing anyone to run anything without password and then write a shitty web UI in front of that. That particular problem is not doas’s (or sudo’s) fault, just as cutting myself with the kitchen knife isn’t the knife’s fault.

                                                              reference your list of 39 Ubuntu vulnerabilities. 39 > 2, Q.E.D.

                                                              Yes, sudo has had more issues in total; I never said it doesn’t. It’s just a lot lower than what you said, and quite a number are very low-impact, so I just disputed the implication that sudo is a security nightmare waiting to happen: it’s track record isn’t all that bad. As always, more features come with more (security) bugs, but use cases do need solving somehow. As I mentioned, it’s a trade-off.

                                                              sudo, on the other hand, can be made much simpler and still address its most common use-cases, as demonstrated by doas’s evident utility

                                                              We already agreed on this yesterday on HN, which I repeated here as well; all I’m adding is “but sudo is still useful, as it solves many more use cases” and “sudo isn’t that bad”.

                                                              Interesting thing to note: sudo was removed from OpenBSD by millert@openbsd.org; who is also the sudo maintainer. I think he’ll agree that “sudo is too complex for it to the default”, which we already agree on, but not that sudo is “too complex to exist”, which is where we don’t agree.

                                                              Could sudo be simpler or better architectured to contain its complexity? Maybe. I haven’t looked at the source or use cases in-depth, and I’m not really qualified to make this judgement.

                                                      2. 5

                                                        I think arguing against complexity is one of the core principles of UNIX philosophy, and it’s gotten us quite far on the operating system front.

                                                        If simplicity was used in sudo, this particular vulnerability would not have been possible to trigger it: why have sudoedit in the first place, which just implies the -e flag? This statement is a guarantee.

                                                        If it would’ve ditched C, there is no guarantee that this issue wouldn’t have happened.

                                                      3. 2

                                                        Did you even look at that list? Most of those are not sudo vulnerabilities but issues in sudo configurations distros ship with.

                                                        If even the distros can’t understand the configuration well enough to get it right, what hope do I have?

                                                      4. 16

                                                        OK maybe here’s a more specific discussion point:

                                                        There can be logic bugs in basically any language, of course. However, the following classes of bugs tend to be steps in major exploits:

                                                        • Bounds checking issues on arrays
                                                        • Messing around with C strings at an extremely low level

                                                        It is hard to deny that, in a universe where nobody ever messed up those two points, there are a lot less nasty exploits in the world in systems software in particular.

                                                        Many other toolchains have decided to make the above two issues almost non-existent through various techniques. A bunch of old C code doesn’t handle this. Is there not something that can be done here to get the same productivity and safety advantages found in almost every other toolchain for tools that form the foundation of operating computers? Including a new C standard or something?

                                                        I can have a bunch of spaghetti code in Python, but turning that spaghetti into “oh wow argv contents ran over some other variables and messed up the internal state machine” is a uniquely C problem, but if everyone else can find solutions, I feel like C could as well (including introducing new mechanisms to the language. We are not bound by what is printed in some 40-year-old books, and #ifdef is a thing).

                                                        EDIT: forgot to mention this, I do think that sudo is a bit special given that its default job is to take argv contents and run them. I kinda agree that sudo is a bit special in terms of exploitability. But hey, the logic bugs by themselves weren’t enough to trigger the bug. When you have a multi-step exploit, anything on the path getting stopped is sufficient, right?

                                                        1. 14

                                                          +1. Lost in the noise of “but not all CVEs…” is the simple fact that this CVE comes from an embarrassing C string fuckup that would be impossible, or at least caught by static analysis, or at very least caught at runtime, in most other languages. If “RWIIR” is flame bait, then how about “RWIIP” or at least “RWIIC++”?

                                                          1. 1

                                                            I be confused… what does the P in RWIIP mean?

                                                            1. 3

                                                              Pascal?

                                                              1. 1

                                                                Python? Perl? Prolog? PL/I?

                                                              2. 2

                                                                Probably Python, given the content of the comment by @rtpg. Python is also memory-safe, while it’s unclear to me whether Pascal is (a quick search reveals that at least FreePascal is not memory-safe).

                                                                Were it not for the relative (accidental, non-feature-providing) complexity of Python to C, I would support RWIIP. Perhaps Lua would be a better choice - it has a tiny memory and disk footprint while also being memory-safe.

                                                                1. 2

                                                                  Probably Python, given the content of the comment by @rtpg. Python is also memory-safe, while it’s unclear to me whether Pascal is (a quick search reveals that at least FreePascal is not memory-safe).

                                                                  That’s possibly it.

                                                                  Perhaps Lua would be a better choice - it has a tiny memory and disk footprint while also being memory-safe.

                                                                  Not to mention that Lua – even when used without LuaJIT – is simply blazingly fast compared to other scripting languages (Python, Perl, &c)!

                                                                  For instance, see this benchmark I did sometime ago: https://0x0.st/--3s.txt. I had implemented Ackermann’s function in various languages (the “./ack” file is the one in C) to get a rough idea on their execution speed, and lo and behold Lua turned out to be second only to the C implementation.

                                                          2. 15

                                                            I agree that rewriting things in Rust is not always the answer, and I also agree that simpler software makes for more secure software. However, I think it is disingenuous to compare the overall CVE count for the two programs. Would you agree that sudo is much more widely installed than doas (and therefore is a larger target for security researchers)? Additionally, most of the 140 CVEs linked were filed before October 2015, which is when doas was released. Finally, some of the linked CVEs aren’t even related to code vulnerabilities in sudo, such as the six Quest DR Series Disk Backup CVEs (example).

                                                            1. 4

                                                              I would agree that sudo has a bigger target painted on its back, but it’s also important to acknowledge that it has a much bigger back - 100× bigger. However, I think the comparison is fair. doas is the default in OpenBSD and very common in NetBSD and FreeBSD systems as well, which are at the heart of a lot of high-value operations. I think it’s over the threshold where we can consider it a high-value target for exploitation. We can also consider the kinds of vulnerabilities which have occured internally within each project, without comparing their quantity to one another, to characterize the sorts of vulnerabilities which are common to each project, and ascertain something interesting while still accounting for differences in prominence. Finally, there’s also a bias in the other direction: doas is a much simpler tool, shipped by a team famed for its security prowess. Might this not dissuade it as a target for security researchers just as much?

                                                              Bonus: if for some reason we believed that doas was likely to be vulnerable, we could conduct a thorough audit on its 500-some lines of code in an hour or two. What would the same process look like for sudo?

                                                              1. -1

                                                                but it’s also important to acknowledge that it has a much bigger back - 100× bigger.

                                                                Sorry but I miss the mass of users pretesting on the streets for tools that have 100x code compare to other tools providing similar functionality.

                                                                1. 10

                                                                  What?

                                                            2. 10

                                                              So you’re saying that 50% of the CVEs in doas would have been prevented by writing it in Rust? Seems like a good reason to write it in Rust.

                                                              1. 11

                                                                Another missing point is that Rust is only one of many memory safe languages. Sudo doesn’t need to be particularly performant or free of garbage collection pauses. It could be written in your favorite GCed language like Go, Java, Scheme, Haskell, etc. Literally any memory safe language would be better than C for something security-critical like sudo, whether we are trying to build a featureful complex version like sudo or a simpler one like doas.

                                                                1. 2

                                                                  Indeed. And you know, Unix in some ways have been doing this for years anyway with Perl, python and shell scripts.

                                                                  1. 2

                                                                    I’m not a security expert, so I’m be happy to be corrected, but if I remember correctly, using secrets safely in a garbage collected language is not trivial. Once you’ve finished working with some secret, you don’t necessarily know how long it will remain in memory before it’s garbage collected, or whether it will be securely deleted or just ‘deallocated’ and left in RAM for the next program to read. There are ways around this, such as falling back to manual memory control for sensitive data, but as I say, it’s not trivial.

                                                                    1. 2

                                                                      That is true, but you could also do the secrets handling in a small library written in C or Rust and FFI with that, while the rest of your bog-standard logic not beholden to the issues that habitually plague every non-trivial C codebase.

                                                                      1. 2

                                                                        Agreed.

                                                                        Besides these capabilities, ideally a language would also have ways of expressing important security properties of code. For example, ways to specify that a certain piece of data is secret and ensure that it can’t escape and is properly overwritten when going out of scope instead of simply being dropped, and ways to specify a requirement for certain code to use constant time to prevent timing side channels. Some languages are starting to include things like these.

                                                                        Meanwhile when you try to write code with these invariants in, say, C, the compiler might optimize these desired constraints away (overwriting secrets is a dead store that can be eliminated, the password checker can abort early when the Nth character of the hash is wrong, etc) because there is no way to actually express those invariants in the language. So I understand that some of these security-critical things are written in inline assembly to prevent these problems.

                                                                        1. 1

                                                                          overwriting secrets is a dead store that can be eliminated

                                                                          I believe that explicit_bzero(3) largely solves this particular issue in C.

                                                                          1. 1

                                                                            Ah, yes, thanks!

                                                                            It looks like it was added to glibc in 2017. I’m not sure if I haven’t looked at this since then, if the resources I was reading were just not up to date, or if I just forgot about this function.

                                                                2. 8

                                                                  I do think high complexity is the source of many problems in sudo and that doas is a great alternative to avoid many of those issues.

                                                                  I also think sudo will continue being used by many people regardless. If somebody is willing to write an implementation in Rust which might be just as complex but ensures some level of safety, I don’t see why that wouldn’t be an appropriate solution to reducing the attack surface. I certainly don’t see why we should avoid discussing Rust just because an alternative to sudo exists.

                                                                  1. 2

                                                                    Talking about Rust as an alternative is missing the forest for the memes. Rust is a viral language (in the sense of internet virality), and a brain worm that makes us all want to talk about it. But in actual fact, C is not the main reason why anything is broken - complexity is. We could get much more robust and reliable software if we focused on complexity, but instead everyone wants to talk about fucking Rust. Rust has its own share of problems, chief among them its astronomical complexity. Rust is not a moral imperative, and not even the best way of solving these problems, but it does have a viral meme status which means that anyone who sees through its bullshit has to proactively fend off the mob.

                                                                    1. 32

                                                                      But in actual fact, C is not the main reason why anything is broken - complexity is.

                                                                      Offering opinions as facts. The irony of going on to talk about seeing through bullshit.

                                                                      1. 21

                                                                        I don’t understand why you hate Rust so much but it seems as irrational as people’s love for it. Rust’s main value proposition is that it allows you to write more complex software that has fewer bugs, and your point is that this is irrelevant because the software should just be less complex. Well I have news for you, software is not going to lose any of its complexity. That’s because we want software to do stuff, the less stuff it does the less useful it becomes, or you have to replace one tool with two tools. The ecosystem hasn’t actually become less complex when you do that, you’re just dividing the code base into two chunks that don’t really do what you want. I don’t know why you hate Rust so much to warrant posting anywhere the discussion might come up, but I would suggest if you truly cannot stand it that you use some of your non-complex software to filter out related keywords in your web browser.

                                                                        1. 4

                                                                          Agree with what you’ve wrote, but just to pick at a theme that’s bothering me on this thread…

                                                                          I don’t understand why you hate Rust so much but it seems as irrational as people’s love for it.

                                                                          This is obviously very subjective, and everything below is anecdotal, but I don’t agree with this equivalence.

                                                                          In my own experience, everyone I’ve met who “loves” or is at least excited about rust seems to feel so for pretty rational reasons: they find the tech interesting (borrow checking, safety, ML-inspired type system), or they enjoy the community (excellent documentation, lots of development, lots of online community). Or maybe it’s their first foray into open source, and they find that gratifying for a number of reasons. I’ve learned from some of these people, and appreciate the passion for what they’re doing. Not to say they don’t exist, but I haven’t really seen anyone “irrationally” enjoy rust - what would that mean? I’ve seen floating around a certain spiteful narrative of the rust developer as some sort of zealous online persona that engages in magical thinking around the things rust can do for them, but I haven’t really seen this type of less-than-critical advocacy any more for rust than I have seen for other technologies.

                                                                          On the other hand I’ve definitely seen solid critiques of rust in terms of certain algorithms being tricky to express within the constraints of the borrow checker, and I’ve also seen solid pushback against some of the guarantees that didn’t hold up in specific cases, and to me that all obviously falls well within the bounds of “rational”. But I do see a fair amount of emotionally charged language leveled against not just rust (i.e. “bullshit” above) but the rust community as well (“the mob”), and I don’t understand what that’s aiming to accomplish.

                                                                          1. 3

                                                                            I agree with you, and I apologize if it came across that I think rust lovers are irrational - I for one am a huge rust proselytizer. I intended for the irrationality I mentioned to be the perceived irrationality DD attributes to the rust community

                                                                            1. 2

                                                                              Definitely no apology needed, and to be clear I think the rust bashing was coming from elsewhere, I just felt like calling it to light on a less charged comment.

                                                                            2. 1

                                                                              I think the criticism isn’t so much that people are irrational in their fondness of Rust, but rather that there are some people who are overly zealous in their proselytizing, as well as a certain disdain for everyone who is not yet using Rust.

                                                                              Here’s an example comment from the HN thread on this:

                                                                              Another question is who wants to maintain four decades old GNU C soup? It was written at a different time, with different best practices.

                                                                              In some point someone will rewrite all GNU/UNIX user land in modern Rust or similar and save the day. Until this happens these kind of incidents will happen yearly.

                                                                              There are a lot of things to say about this comment, and it’s entirely false IMO, but it’s not exactly a nice comment, and why Rust? Why not Go? Or Python? Or Zig? Or something else.

                                                                              Here’s another one:

                                                                              Rust is modernized C. You are looking for something that already exists. If C programmers would be looking for tools to help catch bugs like this and a better culture of testing and accountability they would be using Rust.

                                                                              The disdain is palatable in this one, and “Rust is modernized C” really misses the mark IMO; Rust has a vastly different approach. You can consider this a good or bad thing, but it’s really not the only approach towards memory-safe programming languages.


                                                                              Of course this is not representative for the entire community; there are plenty of Rust people that I like and have considerably more nuanced views – which are also expressed in that HN thread – but these comments certainly are frequent enough to give a somewhat unpleasant taste.

                                                                            3. 2

                                                                              Rust’s main value proposition is that it allows you to write more complex software that has fewer bugs

                                                                              I argue that it’s actually that it allows you to write fast software with fewer bugs. I’m not entirely convinced that Rust allows you to manage complexity better than, say, Common Lisp.

                                                                              That’s because we want software to do stuff, the less stuff it does the less useful it becomes

                                                                              Exactly. Software is written for people to use. (technically, only some software - other software (such as demoscenes) is written for the beauty of it, or the enjoyment of the programmer; but in this discussion we only care about the former)

                                                                              The ecosystem hasn’t actually become less complex when you do that

                                                                              Even worse - it becomes more complex. Now that you have two tools, you have two userbases, two websites, two source repositories, two APIs, two sets of file formats, two packages, and more. If the designs of the tools begin to differ substantially, you have significantly more ecosystem complexity.

                                                                              1. 2

                                                                                You’re right about Rust value proposition, I should have added performance to that sentence. Or, I should have just said managed language, because as another commenter pointed out Rust is almost irrelevant to this whole conversation when it comes to preventing these type of CVEs

                                                                              2. 2

                                                                                While I don’t approve of the deliberately inflammatory form of the comments, and don’t agree with the general statement that all complexity is eliminateable, I personally agree that, in this particular case, simplicity > Rust.

                                                                                As a thought experiment, world 1 uses sudo-rs as a default implementation of sudo, while world 2 uses 500 lines of C which is doas. I do think that world 2 would be generally more secure. Sure, it’ll have more segfaults, but fewer logical bugs.

                                                                                I also think that the vast majority of world 2 populace wouldn’t notice the absence of advanced sudo features. To be clear, the small fraction that needs those features would have to install sudo, and they’ll use the less tested implementation, so they will be less secure. But that would be more than offset by improved security of all the rest.

                                                                                Adding a feature to a program always has a cost for those who don’t use this feature. If the feature is obscure, it might be overall more beneficial to have a simple version which is used by the 90% of the people, and a complex for the rest 10%. The 10% would be significantly worse off in comparison to the unified program. The 90% would be slightly better off. But 90% >> 10%.

                                                                                1. 1

                                                                                  The other issue is that it is a huge violation of principle of least privilege. Those other features are fine, but do they really need to be running as root?

                                                                            4. 7

                                                                              Just to add to that: In addition to having already far too much complexity, it seems the sudo developers have a tendency to add even more features: https://computingforgeeks.com/better-secure-new-sudo-release/

                                                                              Plugins, integrated log server, TLS support… none of that are things I’d want in a tool that should be simple and is installed as suid root.

                                                                              (Though I don’t think complexity vs. memory safety are necessarily opposed solutions. You could easily imagine a sudo-alike too that is written in rust and does not come with unnecessary complexity.)

                                                                              1. 4

                                                                                What’s wrong with EBNF and how is it related to security? I guess you think EBNF is something the user shouldn’t need to concern themselves with?

                                                                                1. 6

                                                                                  There’s nothing wrong with EBNF, but there is something wrong with relying on it to explain an end-user-facing domain-specific configuration file format for a single application. It speaks to the greater underlying complexity, which is the point I’m making here. Also, if you ever have to warn your users not to despair when reading your docs, you should probably course correct instead.

                                                                                  1. 2

                                                                                    Rewrite: The point that you made in your original comment is that sudo has too many features (disguising it as a point about complexity). The manpage snippet that you’re referring to has nothing to do with features - it’s a mix between (1) the manpage being written poorly and (2) a bad choice of configuration file format resulting in accidental complexity increase (with no additional features added).

                                                                                  2. 1

                                                                                    EBNF as a concept aside; the sudoers manpage is terrible.

                                                                                  3. 3

                                                                                    Hello, I am here to derail the Rust discussion before it gets started.

                                                                                    I am not sure what you are trying to say, let me guess with runaway complexity.

                                                                                    • UNIX is inherently insecure and it cannot be made secure by any means
                                                                                    • sudo is inherently insecure and it cannot be made secure by any means

                                                                                    Something else maybe?

                                                                                    1. 4

                                                                                      Technically I agree with both, though my arguments for the former are most decidedly off-topic.

                                                                                      1. 5

                                                                                        Taking Drew’s statement at face value: There’s about to be another protracted, pointless argument about rewriting things in rust, and he’d prefer to talk about something more practically useful?

                                                                                        1. 7

                                                                                          I don’t understand why you would care about preventing a protracted, pointless argument on the internet. Seems to me like trying to nail jello to a tree.

                                                                                      2. 3

                                                                                        This is a great opportunity to promote doas. I use it everywhere these days, and though I don’t consider myself any sort of Unix philosophy purist, it’s a good example of “do one thing well”. I’ll call out Ted Unangst for making great software. Another example is signify. Compared to other signing solutions, there is much less complexity, much less attack surface, and a far shallower learning curve.

                                                                                        I’m also a fan of tinyssh. It has almost no knobs to twiddle, making it hard to misconfigure. This is what I want in security-critical software.

                                                                                        Relevant link: Features Are Faults.

                                                                                        All of the above is orthogonal to choice of implementation language. You might have gotten a better response in the thread by praising doas and leaving iron oxide out of the discussion. ‘Tis better to draw flies with honey than with vinegar. Instead, you stirred up the hornets’ nest by preemptively attacking Rust.

                                                                                        PS. I’m a fan of your work, especially Sourcehut. I’m not starting from a place of hostility.

                                                                                        1. 3

                                                                                          If you want programs to be more secure, stable, and reliable, the key metric to address is complexity. Rewriting it in Rust is not the main concern.

                                                                                          Why can’t we have the best of both worlds? Essentially a program copying the simplicity of doas, but written in Rust.

                                                                                          1. 2

                                                                                            Note that both sudo and doas originated in OpenBSD. :)

                                                                                            1. 9

                                                                                              Got a source for the former? I’m pretty sure sudo well pre-dates OpenBSD.

                                                                                              Sudo was first conceived and implemented by Bob Coggeshall and Cliff Spencer around 1980 at the Department of Computer Science at SUNY/Buffalo. It ran on a VAX-11/750 running 4.1BSD. An updated version, credited to Phil Betchel, Cliff Spencer, Gretchen Phillips, John LoVerso and Don Gworek, was posted to the net.sources Usenet newsgroup in December of 1985.

                                                                                              The current maintainer is also an OpenBSD contributor, but he started maintaining sudo in the early 90s, before OpenBSD forked from NetBSD. I don’t know when he started contributing to OpenBSD.

                                                                                              So I don’t think it’s fair to say that sudo originated in OpenBSD :)

                                                                                              1. 1

                                                                                                Ah, looks like I was incorrect. I misinterpreted OpenBSD’s innovations page. Thanks for the clarification!

                                                                                          1. 2

                                                                                            Interesting overview of the Nix approaches, so thanks for sharing. My curiosity has been piqued by all the NixOS posts I’ve been seeing even though I don’t run it myself. Any reports from folks running a bunch of devices like the one from this post on their home network? What are you using them to do?

                                                                                            My home router/firewall/dhcp/ipsec server (atom x86-64) and file/media/proxy/cache/print server (celeron x86-64) are OpenBSD, so when I bought a Beaglebone Black (ARMv7) to toy with, it was fun to go down a rabbit hole pretending I was @tedu (his 2014 post) to learn about diskless(8) and pxeboot(8) and how to netboot via uboot. This ended being pure experimentation since the actual parallelized work I do at home is on a single beefy Linux workstation (hard requirement on Nvidia GPU for now) and I’m not a professional sysadmin. The BBB sits disconnected in a drawer, but the setup lives on as the mere handful of config line changes required to set up tftpd(8) on the file server and point dhcpd(8) to it from the router, so I gained a more complete understanding of those as a neat side effect of experimenting. At some point in the next couple years I’m going to want to play with a RISC-V SoC, but that’s going to mean looking at Linux again unless I magically become competent to write my own drivers.

                                                                                            1. 8

                                                                                              I just converted my last non-NixOS machine yesterday, so I’ll share my experience =]

                                                                                              I currently have 5 machines running NixOS and deployed using NixOps (to a network called ekumen):

                                                                                              • laptop, ThinkPad T14 AMD (odo)
                                                                                              • workstation: Ryzen 9 3900X, 128GB RAM (takver)
                                                                                              • compute stick: quad core Intel Atom, 2GB RAM (efor)
                                                                                              • rpi: 3B+, 1 GB RAM (gvarab)
                                                                                              • chromebox: i7, 16GB RAM (mitis)

                                                                                              I set up the workstation and chromebox as remote builders for all systems, just as @steinuil did in the post. I’m using the rpi for running Jellyfin (music) and Nextcloud (for sharing calendars and files with my spouse), and setting up the chromebox to be an IPFS node for sharing research data. The laptop and workstation are using home-manager for syncing my dev environment configurations, but I do most of the dev/data analysis in the workstation (which has gigabit connections to the internet), and while the laptop is often more than enough for dev, my home connection is way too slow for anything network-intensive (so, it serves as a glorified SSH client =P)

                                                                                              They are all wired together using zerotier, and services running in the machines are bound to the zerotier interface, which ends up creating a pretty nice distributed LAN.

                                                                                              I don’t have my configs in public (booo!), because I’ve not been too good on maintaining secrets out of the configs. But @cadey posts are a treasure trove of good ideas, and I also enjoyed this post and accompanying repo as sources of inspiration.

                                                                                              1. 1

                                                                                                I don’t really see the value nixops provides over nixos-rebuild which can work over ssh.

                                                                                                1. 1

                                                                                                  That’s a fair point. Part of using nixops was about exploring how to use it later for other kinds of deployment (clouds), and it is a bit excessive for my use case (especially since I use nixops to deploy locally in the laptop =P).

                                                                                                  A lot of my nix experience so far is seeing multiple implementations of similar concepts, but I also feel like I can refactor and try other approaches without borking my systems (too much).

                                                                                              2. 2

                                                                                                On the Pi from the post I run:

                                                                                                • syncthing
                                                                                                • Navidrome so I can listen to my music library on my phone
                                                                                                • twkwk, a small program that serves my TiddlyWiki instance
                                                                                                • synapse, a torrent client which is actually the Rust program I mentioned in the post
                                                                                                • some SMB shares with Samba that serve the two drives I use for torrents and music
                                                                                              1. 3

                                                                                                When I think of the phrase “spooky action at a distance” with respect to programming, the thing that always comes to my mind is mutable state. I know of no better analogy within programming to quantum mechanics’ “spooky action at a distance” than mutable state, though admittedly I know next to nothing about quantum mechanics. Mutable objects in programming seem a lot like objects that have been “quantum entangled”. I think about this a lot when I have reason to share a piece of mutable state between two objects, such as when writing a unification-based type system (which one of my current projects includes). I find it interesting to think about how the type of some expression, as a type variable, can propagate down different branches of a program tree. Then at some point the type checker unifies the type on one branch to something (more) fully specified, and suddenly, spookily, the type of expressions in a distant branch are similarly specified.

                                                                                                Not that I think mutable state is inherently bad – it’s a great engineering tool that is often overly maligned by functional programming purists. But I do think, just like good design of countless other things in programming, exactly how to use mutable state with clarity rather than confusion requires good taste and judgment. (This is not a claim that I necessarily have the best taste and judgment.)

                                                                                                Now I’d like to nitpick a statement from the OP, even though others are commenting on the same statement:

                                                                                                x + y

                                                                                                If this were written in C, without knowing anything other than the fact that this code compiles correctly, I can tell you that x and y are numeric types, and the result is their sum.

                                                                                                In C, without using a particular compiler that specifies particular semantics beyond the standard, you can not know the result (or even the behavior of the surrounding code or entire program!) of a + b without knowing the dynamic state of the running program, because a + b can result in undefined behavior. There is no programming language more spooky than C.

                                                                                                1. 2

                                                                                                  I don’t think it’s mutable state by itself that’s the problem, it’s aliased mutable state. In C, I can write code like this:

                                                                                                  int *a = &foo;
                                                                                                  int *b = a + 42;
                                                                                                  b[2] = 12;
                                                                                                  

                                                                                                  And this changes the value of a[54], even though I never used the name a in my mutation. That’s action at a distance and, to me, a good language should provide some static type information to tell you when things might be aliased (neither C nor C++ does a good job here).

                                                                                                  Aliasing between concurrent execution contexts is the worst case of this. In C, there’s no protection against this at all and (worse) the language says it’s undefined behaviour if two threads concurrently update a variable and it isn’t _Atomic qualified. Shared mutable state is the worst possible kind of spooky action at a distance: you can step through a thread one instruction at a time and still see a result that you couldn’t explain from looking at that thread’s code. This is why Verona makes it impossible to express concurrently mutable state.

                                                                                                  1. 1

                                                                                                    I agree, without mutable state a function can be efficient, or inefficient, but never really spooky.

                                                                                                    1. 1

                                                                                                      Regarding your mention of unification state propagating through different branches of the program tree: you may be interested in this paper, which defines a type system that’s strictly compositional (the type of an expression is based entirely on the type of its subexpressions).

                                                                                                    1. 9

                                                                                                      If this were written in C, without knowing anything other than the fact that this code compiles correctly, I can tell you that x and y are numeric types, and the result is their sum. I can even make an educated guess about the CPU instructions which will be generated to perform this task.

                                                                                                      x could be a pointer and y could be an int. A pointer isn’t a numeric type. On most architectures, it kind of is numeric; it stores an address – which is just a numeric index into memory. FWIW I never fully grokked pointers until I learned assembly.

                                                                                                      C hides a lot of details too. That’s what programming languages at a higher level than assembly are all about: abstracting over details.

                                                                                                      Regardless of what language you write in, it’s good to be aware of the performance characteristics of your code. And it’s totally possible to cultivate that sort of awareness, even in the face of things like operator overloading. Start by learning some assembly.

                                                                                                      1. 7

                                                                                                        Not to mention that even if they’re numeric types, then “the result is their sum” can still not be true either - IIRC:

                                                                                                        • if the types are unsigned, this line could be an overflow and thus the result would be sum modulo;
                                                                                                        • even worse, if the types are signed, this line could be an undefined behavior, and thus a spooky eldritch Cthulhu action crawling slowly throughout your app and devouring any logic.

                                                                                                        Also, the malloc line does not check the returned value, so:

                                                                                                        • in case you run close to an out-of-memory condition, the strcpy and strcat could overwrite memory, also doing spooky eldritch Cthulhu corruption action;
                                                                                                        • in case x or y come from your users, they could be maliciously crafted such that strlen could overflow int, leading again to spooky eldritch Cthulhu corruption action, security breach etc. etc. (I think; not 100% sure if some other effect doesn’t accidentally neutralize this, I don’t care enough about C anymore to try and track precisely what would happen there in such case).
                                                                                                        1. 1

                                                                                                          in case you run close to an out-of-memory condition, the strcpy and strcat could overwrite memory, also doing spooky eldritch Cthulhu corruption action;

                                                                                                          if there is no memory, it returns NULL - this is probably just gonna segfault and not corrupt anything.

                                                                                                        2. 5

                                                                                                          I think this comment, and @akavel’s follow-up, are both missing the point. Yes, it’s more complex than my brief summary affords. However, the action is well-defined and constrained by the specification, and you can learn what it entails, then apply that knowledge to your understanding of all C programs. On the contrary, for languages which have operator overloading, you can never fully understand the behavior of the + operator without potentially consulting project-specific code.

                                                                                                          1. 3

                                                                                                            I totally get the point of “spooky action at a distance”, and totally agree with it; I learnt of this idea in programming quite some time ago and do look at it similarly since. I also honestly appreciate the look at #define in C as a feature that can be said to exhibit a similar mechanism; I never thought of it like this, and I find it a really cool and interesting take.

                                                                                                            Yet, I still do find the claims about C’s simplicity that this article makes highly misleading. Especially in this context of subtle hidden difficulties. If it at least provided some disclaimer footnote (or whatever else form given gemini), I could swallow this. But as shown in the comments, some of the sentences are currently just plain wrong.

                                                                                                            Let me phrase this differently: I believe this could be a great article. I could recommend it to others. However, currently I see it as, looking at a whole, an okay-ish article. Some good points, some false claims. I don’t plan to advertise it to others, given the dangerous and unfortunately seductive ideas it also conveys (esp. to non-expert or just would-be C programmers) “piggybacked” with the main argument (e.g. that “C is easy and simple”). I lament that the article is not better. And I’m painfully aware I need to actually warn some people against naively succumbing to the overt simplifications presented in this article. Because I’ve seen so many unaware and sincerely gullible people, and myself have been unaware for ohhhh so long

                                                                                                            1. 1

                                                                                                              I don’t think the omission is consequential to the point of the article. In fact, it criticizes C for its role in this “spooky action at a distance”. The point is not to explain how C works.

                                                                                                              1. 2

                                                                                                                Given that this reply basically rehashes parts of what both of us already wrote above, and I apparently read some parts of the article differently than you, I’m assuming neither of us nor anyone else will probably gain anything from any further discussion by us on this matter.

                                                                                                            2. 1

                                                                                                              What C code seemingly free of function calls does depends on platform as well, for instance see:

                                                                                                              https://godbolt.org/z/jsWqzM

                                                                                                              Here binary operations on 64-bit operands compiled as 32-bit code may result in function calls. The other example is initializing a structure of a certain size, which may also result in a function call.

                                                                                                            3. 1

                                                                                                              Right. Try this for fun,

                                                                                                              cat <<'.'|cc -x c - && ./a.out && rm a.out
                                                                                                              #include<stdio.h>
                                                                                                              #define P(T) do{T*x=0;printf("%lx\n",(unsigned long)(x+y));}while(0)
                                                                                                              int main(int c,char**v){int y=1;P(char);P(short);P(int);P(long);P(void);return 0;}
                                                                                                              .
                                                                                                              
                                                                                                            1. 23

                                                                                                              I used to think this was the case until I realized that Google funds Firefox through noblesse oblige, and so all the teeth-gnashing over “Google owns the Internet” is still true whether you use Chrome directly or whether you use Firefox. The only real meaningful competition in browsers is from Apple (God help us.) Yes, Apple takes money from Google too, but they don’t rely on Google for their existence.

                                                                                                              I am using Safari now, which is… okay. The extension ecosystem is much less robust but I have survived. I’m also considering Brave, but Chromium browsers just gulp down the battery in Mac OS so I’m not totally convinced there yet.

                                                                                                              Mozilla’s recent political advocacy has also made it difficult for me to continue using Firefox.

                                                                                                              1. 18

                                                                                                                I used to think this was the case until I realized that Google funds Firefox through noblesse oblige, and so all the teeth-gnashing over “Google owns the Internet” is still true whether you use Chrome directly or whether you use Firefox.

                                                                                                                I’m not sure the premise is true. Google probably wants to have a practical monopoly that does not count as a legal monopoly. This isn’t an angelic motive, but isn’t noblesse oblige.

                                                                                                                More importantly, the conclusion doesn’t follow–at least not 100%. Money has a way of giving you control over people, but it can be imprecise, indirect, or cumbersome. I believe what Google and Firefox have is a contract to share revenue with Firefox for Google searches done through Firefox’s url bar. If Google says “make X, Y and Z decisions about the web or you’ll lose this deal”, that is the kind of statement antitrust regulators find fascinating. Since recent years have seen increased interest in antitrust, Google might not feel that they can do that.

                                                                                                                1. 9

                                                                                                                  Yes, I agree. It’s still bad that most of Mozilla’s funding comes from Google, but it matters that Mozilla is structured with its intellectual property owned by a non-profit. That doesn’t solve all problems, but it creates enough independence that, for example, Firefox is significantly ahead of Chrome on cookie-blocking functionality - which very much hits Google’s most important revenue stream.

                                                                                                                  1. 4

                                                                                                                    Google never has to say “make X, Y and Z decisions about the web or you’ll lose this deal,” with or without the threat of antitrust regulation. People have a way of figuring out what they have to do to keep their job.

                                                                                                                  2. 17

                                                                                                                    I’m tired of the Pocket suggested stories. They have a certain schtick to them that’s hard to pin down precisely but usually amounts to excessively leftist, pseudo-intellectual clickbait: “meat is the privilege of the west and needs to stop.”

                                                                                                                    I know you can turn them off.

                                                                                                                    I’m arguing defaults matter, and defaults that serve to distract with intellectual junk is not great. At least it isn’t misinformation, but that’s not saying much.

                                                                                                                    Moving back to Chrome this year because of that, along with some perf issues I run into more than I’d like. It’s a shame, I wanted to stop supporting Google, but the W3C has succeeded in creating a standard so complex that millions of dollars are necessary to adequately fund the development of a performant browser.

                                                                                                                    1. 2

                                                                                                                      Moving back to Chrome this year because of that, along with some perf issues I run into more than I’d like. It’s a shame, I wanted to stop supporting Google, but the W3C has succeeded in creating a standard so complex that millions of dollars are necessary to adequately fund the development of a performant browser.

                                                                                                                      In case you haven’t heard of it, this might be worth checking out: https://ungoogled-software.github.io/

                                                                                                                      1. 1

                                                                                                                        Except as of a few days ago Google is cutting off access to certain APIs like Sync that Chromium was using.

                                                                                                                        1. 1

                                                                                                                          Straight out of the Android playbook

                                                                                                                    2. 4

                                                                                                                      Mozilla’s recent political advocacy has also made it difficult for me to continue using Firefox.

                                                                                                                      Can you elaborate on this? I use FF but have never delved into their politics.

                                                                                                                      1. 16

                                                                                                                        My top of mind example: https://blog.mozilla.org/blog/2021/01/08/we-need-more-than-deplatforming/

                                                                                                                        Also: https://blog.mozilla.org/blog/2020/07/13/sustainability-needs-culture-change-introducing-environmental-champions/ https://blog.mozilla.org/blog/2020/06/24/immigrants-remain-core-to-the-u-s-strength/ https://blog.mozilla.org/blog/2020/06/24/were-proud-to-join-stophateforprofit/

                                                                                                                        I’m not trying to turn this into debating specifically what is said in these posts but many are just pure politics, which I’m not interested in supporting by telling people to use Firefox. My web browser doesn’t need to talk about ‘culture change’ or systemic racism. Firefox also pushes some of these posts to the new tab page, by default, so it’s not like you can just ignore their blog.

                                                                                                                        1. 6

                                                                                                                          I’m started to be afraid that being against censorship is enough to get you ‘more than de-platformed’.

                                                                                                                            1. 10

                                                                                                                              Really? I feel like every prescription in that post seems reasonable; increase transparency, make the algorithm prioritize factual information over misinformation, research the impact of social media on people and society. How could anyone disagree with those points?

                                                                                                                              1. 17

                                                                                                                                You’re right, how could anyone disagree with the most holy of holies, ‘fact checkers’?

                                                                                                                                Here’s a great fact check: https://www.politifact.com/factchecks/2021/jan/06/ted-cruz/ted-cruzs-misleading-statement-people-who-believe-/

                                                                                                                                The ‘fact check’ is a bunch of irrelevant information about how bad Ted Cruz and his opinions are, before we get to the meat of the ‘fact check’ which is, unbelievably, “yes, what he said is true, but there was also other stuff he didn’t say that we think is more important than what he did!”

                                                                                                                                Regardless of your opinion on whether this was a ‘valid’ fact check or not, I don’t want my web browser trying to pop up clippy bubbles when I visit a site saying “This has been officially declared by the Fact Checkers™ as wrongthink, are you sure you’re allowed to read it?” I also don’t want my web browser marketer advocating for deplatforming (“we need more than deplatforming suggests that deplatforming should still be part of the ‘open’ internet.) That’s all.

                                                                                                                                1. 15

                                                                                                                                  a bunch of irrelevant information about how bad Ted Cruz and his opinions are

                                                                                                                                  I don’t see that anywhere. It’s entirely topical and just some context about what Cruz was talking about.

                                                                                                                                  the meat of the ‘fact check’ which is, unbelievably, “yes, what he said is true, but there was also other stuff he didn’t say that we think is more important than what he did!”

                                                                                                                                  That’s not what it says at all. Anyone can cherry-pick or interpret things in such a way that makes their statement “factual”. This is how homeopaths can “truthfully” point at studies which show an effect in favour of homeopathy. But any fact check worth its salt will also look at the overwhelming majority of studies that very clearly demonstrate that homeopathy is no better than a placebo, and therefore doesn’t work (plus, will point out that the proposed mechanisms of homeopathy are extremely unlikely to work in the first place, given that they violate many established laws of physics).

                                                                                                                                  The “39% of Americans … 31% of independents … 17% of Democrats believe the election was rigged” is clearly not supported by any evidence, and only by a tenuous interpretation of a very limited set of data. This is a classic case of cherry-picking.

                                                                                                                                  I hardly ever read politifact, but if this is really the worst fact-check you can find then it seems they’re not so bad.

                                                                                                                                  1. 7

                                                                                                                                    This article has a few more examples of bad fact checks:

                                                                                                                                    https://greenwald.substack.com/p/instagram-is-using-false-fact-checking

                                                                                                                                  2. 7

                                                                                                                                    Media fact-checkers are known to be biased.

                                                                                                                                    [Media Matters lobby] had to make us think that we needed a third party to step in and tell us what to think and sort through the information … The fake news effort, the fact-checking, which is usually fake fact-checking, meaning it’s not a genuine effort, is a propaganda effort … We’ve seen it explode as we come into the 2020 election, for much the same reason, whereby, the social media companies, third parties, academic institutions and NewsGuard … they insert themselves. But of course, they’re all backed by certain money and special interests. They’re no more in a position to fact-check than an ordinary person walking on the street … — Sharyl Attkisson on Media Bias, Analysis by Dr. Joseph Mercola

                                                                                                                                    Below is a list of known rebuttals of some “fact-checkers”.

                                                                                                                                    Politifact

                                                                                                                                    • I wanted to show that these fact-checkers just lie, and they usually go unchecked because most people don’t have the money, don’t have the time, and don’t have the platform to go after them — and I have all three” — Candace Owens Challenges Fact-Checker, And Wins

                                                                                                                                    Full fact (fullfact.org)

                                                                                                                                    Snopes

                                                                                                                                    Associated Press (AP)

                                                                                                                                    • Fact-checking was devised to be a trusted way to separate fact from fiction. In reality, many journalists use the label “fact-checking” as a cover for promoting their own biases. A case in point is an Associated Press (AP) piece headlined “AP FACT-CHECK: Trump’s inaccurate boasts on China travel ban,” which was published on March 26, 2020 and carried by many news outlets.” — Propaganda masquerading as fact-checking

                                                                                                                                    Politico

                                                                                                                                    1. 4

                                                                                                                                      I’m interested in learning about the content management systems that these fact checker websites use to effectively manage large amounts of content with large groups of staff. Do you have any links about that?

                                                                                                                                      1. 3

                                                                                                                                        The real error is to imply that “fact checkers” are functionally different from any other source of news/journalism/opinion. All such sources are a collection of humans. All humans have bias. Many such collections of humans have people that are blind to their own bias, or suffer a delusion of objectivity.

                                                                                                                                        Therefore the existence of some rebuttals to a minuscule number of these “fact checks” (between 0 and 1% of all “fact checks”) should not come as a surprise to anyone. Especially when the rebuttals are published by other news/journalism/opinion sources that are at least as biased and partisan as the fact checkers they’re rebutting.

                                                                                                                                        1. 1

                                                                                                                                          The real error is to imply that “fact checkers” are functionally different from any other source of news/journalism/opinion.

                                                                                                                                          Indeed they aren’t that different. Fact-checkers inherit whatever bias that is already present in mainstream media, which itself is a well-documented fact, as the investigative journalist Sharyl Atkisson explored in her two books:

                                                                                                                                          • The Smear exposes and focuses on the multi-billion dollar industry of political and corporate operatives that control the news and our info, and how they do it.
                                                                                                                                          • Slanted looks at how the operatives moved on to censor info online (and why), and has chapters dissecting the devolution of NYT and CNN, recommendations where to get off narrative news, and a comprehensive list of media mistakes.
                                                                                                                                  3. 5

                                                                                                                                    After reading that blog post last week I switched away from Firefox. It will lead to the inevitable politicization of a web browser where the truthfulness of many topics is filtered through a very left-wing, progressive lens.

                                                                                                                                    1. 22

                                                                                                                                      I feel like “the election wasn’t stolen” isn’t a left- or right-wing opinion. It’s just the truth.

                                                                                                                                      1. 15

                                                                                                                                        To be fair, I feel like the whole idea of the existence of an objective reality is a left-wing opinion right now in the US.

                                                                                                                                        1. 5

                                                                                                                                          There are many instances of objective reality which left-wing opinion deems problematic. It would be unwise to point them out on a public forum.

                                                                                                                                          1. 8

                                                                                                                                            I feel like you have set up a dilemma for yourself. In another thread, you complain that we are headed towards a situation where Lobsters will no longer be a reasonable venue for exploring inconvenient truths. However, in this thread, you insinuate that Lobsters already has become unreasonable, as an excuse for avoiding giving examples of such truths. Which truths are being silenced by Lobsters?

                                                                                                                                            Which truths are being silenced by Mozilla? Keep in mind that the main issue under contention in their blog post is whether a privately-owned platform is obligated to repeat the claims of a politician, particularly when those claims would undermine democratic processes which elect people to that politician’s office; here, there were no truths being silenced, which makes the claim of impending censorship sound like a slippery slope.

                                                                                                                                            1. 4

                                                                                                                                              Yeah but none that are currently fomenting a coup in a major world power.

                                                                                                                                        2. 16

                                                                                                                                          But… Mozilla has been inherently political the whole way. The entire Free Software movement is incredibly political. Privacy is political. Why is “social media should be more transparent and try to reduce the spread of blatant misinformation” where you draw the line?

                                                                                                                                          1. 5

                                                                                                                                            That’s not where I draw the line. We appear to be heading towards a Motte and Bailey fallacy where recent events in the US will be used as justification to clamp down on other views and opinions that left-wing progressives don’t approve of (see some of the comments on this page about ‘fact checkers’)

                                                                                                                                            1. 7

                                                                                                                                              In this case though, the “views and opinions that left-wing progressives don’t approve of” are the ideas of white supremacy and the belief that the election was rigged. Should those not be “clamped down” on? (I mean, it’s important to be able to discuss whether the election was rigged, but not when it’s just a president who doesn’t want to accept a loss and has literally no credible evidence of any kind.)

                                                                                                                                              1. 2

                                                                                                                                                I mentioned the Motte and Bailey fallacy being used and you bring up ‘white supremacy’ in your response! ‘White Supremacy’ is the default Motte used by the progressive left. The Bailey being a clamp down on much more contentious issues. Its this power to clamp down on the more contentious issues that I object to.

                                                                                                                                                1. 6

                                                                                                                                                  So protest clamp downs on things you don’t want to see clamp downs on, and don’t protest clamp downs on things you feel should be clamped down on? We must be able to discuss and address real issues, such as the spread of misinformation and discrimination/supremacy.

                                                                                                                                                  But that’s not even super relevant to the article in question. Mozilla isn’t even calling for censoring anyone. It’s calling for a higher degree of transparency (which none of us should object to) and for the algorithm to prioritize factual information over misinformation (which everyone ought to agree with in principle, though we can criticize specific ways to achieve it).

                                                                                                                                                  1. 4

                                                                                                                                                    We are talking past each other in a very unproductive way.

                                                                                                                                                    The issue I have is with what you describe as “…and for the algorithm to prioritize factual information over misinformation”

                                                                                                                                                    Can you not see the problem when the definition of ‘factual information’ is in the hands of a small group of corporations from the West Coast of America? Do you think that the ‘facts’ related to certain hot-button issues will be politically neutral?

                                                                                                                                                    It’s this bias that i object to.

                                                                                                                                                    This American cultural colonialism.

                                                                                                                                                    1. 3

                                                                                                                                                      Can you not see the problem when the definition of ‘factual information’ is in the hands of a small group of corporations from the West Coast of America?

                                                                                                                                                      ReclaimTheNet recently published a very good article on this topic

                                                                                                                                                      https://reclaimthenet.org/former-aclu-head-ira-glasser-explains-why-you-cant-ban-hate-speech/

                                                                                                                                                      1. 3

                                                                                                                                                        That’s an excellent article. Thank you for posting it.

                                                                                                                                                        1. 3

                                                                                                                                                          You’re welcome. You might be interested in my public notes on the larger topic, published here.

                                                                                                                                          2. 3

                                                                                                                                            Out of interest, to which browser did you switch?

                                                                                                                                      2. 2

                                                                                                                                        if possible, try vivaldi, being based on chromium, it will be easiest to switch to f.e. you can install chromium’s extensions in vivaldi. not sure about their osx (which seems to be your use-case), support though, so ymmv.

                                                                                                                                      1. 11

                                                                                                                                        In regards to mention of Nazi deathcamp imagery on 9front’s site, they’ve removed the image in question and updated the page with an explanation:

                                                                                                                                        Once upon a time, khm was searching for rails documentation and accidentally hit the Images link. In the first page of results was a photo of the train tracks of Auschwitz. Its presence among the Ruby on Rails logos was so absurd and out of place that khm memorialized it. It was made in an era before actual Nazis had re-entered the public dialogue, so it felt like Google Image Search was denigrating Ruby on Rails by including this sort of imagery in the results.

                                                                                                                                        Generally speaking, 4chan types read it as an endorsement, which sucks. More recently, people who are not assholes have also begun to read it as an endorsement, which is even more unfortunate. Finally, the people who just get mad about things on social media have begun nesting in it. As a result, this image has been targeted for redaction by the 9front Internet Mob Mollification committee.

                                                                                                                                        Also note the “Nazi punks fuck off” badge has been on 9front’s front page ever since I found out about them.
                                                                                                                                        I don’t think they condone nazisim, or welcome fascists into their community.

                                                                                                                                        1. 9

                                                                                                                                          Also added, for people who didn’t find “Nazi punks fuck off” sufficiently explicit:

                                                                                                                                          ACHTUNG! 9front absolutely and unalterally opposed to racism, sexism, homophobia, transphobia, nationalism, ethnocentrism, religious fundamentalism, and oppressive and coercive power structures of all kinds.

                                                                                                                                          1. 3

                                                                                                                                            For context. When 9front was still on Google Code they added tags like SOAP, cloud, enterprise, oracle (SOAP being the only thing I certainly remember, but you get the gist). They used that the same way and I am pretty sure they weren’t secretly in love with those things.

                                                                                                                                            Yes, it’s easy to call call something tasteless, if your taste is different and matches the taste of majority. 9front all the way through is is doing the opposite, so I think the nazi meme stuff is similar. I mean attaching software to ideologies or vice versa (maybe other than licenses) seems odd.

                                                                                                                                            It’s also not my taste. Attaching political labels to some one based on memes they find funny seems really off though.

                                                                                                                                            There is a more fitting comparison maybe. Nero Burning Rom, which I have seen criticized for being named after a violent dictator and making a joke out of it. It was actually compared with Nazis as well.

                                                                                                                                            I think people should judged by overall sentiment, actions, and what they actually say, not something that manages to cause such a discussion.

                                                                                                                                            Of course I cannot look into someones mind, so certainly also not saying that there isn’t such a reason behind it, but since it can easily be interpreted as joke, ironically and there is nothing indicating more than that I would go by giving the benefit of the doubt.

                                                                                                                                            Yes, stay vigilant, but let’s not turn this into thought police.

                                                                                                                                            1. 4

                                                                                                                                              Yeah, sure…they put up “we’re not -ism or -phobia we swear” and took down the Nazi pr0n. After vocally arguing why they should’t have to and we’ve done nothing wrong and anyone who had a problem with it are “the people who just get mad about things on social media”. Instead of saying “Whoa! We put up a picture of fucking Auschwitz? Hey lemme delete that right now”.

                                                                                                                                              The optics are less than optimal.

                                                                                                                                              1. 7

                                                                                                                                                When I went to the killing fields in Cambodia there was this audio tour where a survivor narrated the place as you walked around. For reference, there were children sticking their hands through the fences trying to beg money from the rich tourists wandering around the mass grave (the grave was one of very many killing fields, it wasn’t a special one or anything, the cambodian genocide was seriously super horrific). Every time it rains more bones will come out of the ground so they have a collection bucket at the exit if you find any bones while walking through. At the entrance there is a monument that is a tower full of human skulls.

                                                                                                                                                The walk features such things as the tree where they killed children by bashing their heads against it (apparently more efficient than using other means) and of course there is a permanent bloodstain and the bark is eroded where the impact would happen.

                                                                                                                                                As I was going through this tour I was thinking “why on earth am I here, this is literally the most horrific thing I’ve ever seen! Who would have the capacity to do this???”

                                                                                                                                                But at the end of the tour the voice begs and pleads: “Please do not allow this atrocity to be forgotten, we must remember that these things have happened so that we do not grow complacent thinking that ‘nobody would do such a thing’. We must never let such a thing happen again.” (paraphrased since I don’t remember what he said verbatim).

                                                                                                                                                This stuck with me and since then I have become better aware of other such events, for example visiting Rwanda you can feel the scar in the culture. “Dancing in the Glory of Monsters” is a book that if you can stomach it (I wasn’t able to) shows you how the Rwandan genocide is still ongoing in the Congo…. Speaking of, I know people from Venezuela and their refugee crysis has grown faster and further than the Syrian one and their current situation is tragic.

                                                                                                                                                These countries are being torn apart by the wealth they hold (and I believe the vultures that we admire for how much money they can stockpile must be held responsible if we want to start doing anything about this - but that is another matter).

                                                                                                                                                However even though I have very strong feelings on the matter and even though I have good friends from Venezuela that are affected by the crysis there, still.. I will not fault you for showing me a picture of Venezuela!

                                                                                                                                                Actually. I’m quite triggered that you consider it immoral to show a picture of Auschwitz. Repressing memories is not a good way to come to terms with reality, when a loved one dies it is better to honor their memory and respect whatever force it was that destroyed them. Rather than acting like nothing ever happened while you desperately try to hold onto your ignorance of the evils in the world.

                                                                                                                                                IF I ever use the word “hate” with all the weight that it holds then I will use it to describe these symptom-treating efforts that ultimately undermine any efforts at real healing.

                                                                                                                                                1. 10

                                                                                                                                                  If someone was to use one of the many horrific images from the Cambodian killing fields as a reaction image comparing different programming languages, would you consider it a good way of honoring the victims?

                                                                                                                                                  1. 2

                                                                                                                                                    No admittedly not, but it’d be a good opportunity to start a reasonable discussion with whoever is responsible (probably a child[ish] person).

                                                                                                                                                    Deciding unilaterally where the line is and then dogmatically shaming is what I take issue with.

                                                                                                                                                    Edit: not advocating infinite patience but maybe one or two iterations of reserved judgment would go a long way towards defusing misunderstandings or having differing values clashing.

                                                                                                                                                  2. 8

                                                                                                                                                    Actually. I’m quite triggered that you consider it immoral to show a picture of Auschwitz

                                                                                                                                                    Context is important here. Showing a picture on a historical/educational website is great, showing it on some edgy website for a fringe OS is completely tasteless.

                                                                                                                                                    1. 2

                                                                                                                                                      Well there was further context with the ruby thing and then inertia took over. Idk it’s obviously not the appropriate context but all it would take is a caption with some platitude and a link to educational resource for it to be suddenly a brave gesture to keep the memory alive… people are fickle.

                                                                                                                                                  3. 3

                                                                                                                                                    You seem very hard to please, context and time changes, they took it down.

                                                                                                                                                    1. 7
                                                                                                                                                      • Puts up a photo of Auschwitz
                                                                                                                                                      • Shouts all over Mastodon how it’s not going to be removed, claiming it “provokes thought instead of dogma” and any opinion to the contrary is “breathless internet pearl-clutching” (lmao sounds like somebody’s thought ain’t being provoked!)
                                                                                                                                                      • Removes it, adds snark towards the author of this article and attributes the removal to mollifying an internet mob

                                                                                                                                                      nah, sounds like not much has changed at all

                                                                                                                                                      1. 3

                                                                                                                                                        The snark isn’t entirely unwarranted, especially considering how the author of this article felt the need to include 2 paragraphs about how the 9front devs are awful, terrible people for not immediately doing exactly what he asked.

                                                                                                                                                        1. 3

                                                                                                                                                          Nah: the author of the article asked a question on Mastodon that most people would have – “why is there nazi imagery associated with this project” – and after one follow up question a 9front dev[1] blew him off as “starting an internet witch hunt”.

                                                                                                                                                          [1]: That’d be you, yes? Typically it’s good form to say “here’s my side of things” so people know you aren’t just a dispassionate observer sharing their opinion.

                                                                                                                                                          1. 4

                                                                                                                                                            (no, that is not me. I’m not involved in 9front at all, besides occasionally looking at it and booting it up on a pi to see how it’s going. I am truly a dispassionate observer here– thanks for trying to root me out though!)

                                                                                                                                                            It is important to look at the tone of the discussion here: the author opens with “I am now used to the FQA being frankly not worth the 1s and 0s it was written in” (a very kind dismissal of someone’s work!) and then immediately poking and prodding about who, exactly, committed the image. It’s understandable that someone would get defensive if that happened. At this point the aftermath is pointless social media bullshit and is probably off-topic to this site.

                                                                                                                                                            1. 1

                                                                                                                                                              (no, that is not me. I’m not involved in 9front at all, besides occasionally looking at it and booting it up on a pi to see how it’s going. I am truly a dispassionate observer here– thanks for trying to root me out though!)

                                                                                                                                                              Fair enough – I thought I caught a whiff of username overlap but there’s really only 26^2 bits of information there for me to key off ;).

                                                                                                                                                              1. 1

                                                                                                                                                                It is important to look at the tone of the discussion here

                                                                                                                                                                If this is really the worst thing you can pull out about what I’ve written online I’d say that’s not bad. Can you provide me an example of where I’m apparently poking and prodding about who, exactly, committed the image? Because the only person I asked about was here with reasons for asking in the question. I didn’t know when I asked but it turns out the info is publicly available anyway.

                                                                                                                                                                the author opens with “I am now used to the FQA being frankly not worth the 1s and 0s it was written in” (a very kind dismissal of someone’s work!)

                                                                                                                                                                No I don’t. This isn’t even in the linked article. It was taken from part of this mastodon conversation between two people. It’s in reference to the state of the FQA where I’d written notes to submit to fill gaps until I saw the image. I find it odd to assert a frustration shared as part of a discussion between two people is a dismissal when the document itself says:

                                                                                                                                                                ACHTUNG! Information provided by this document is UNOFFICIAL and may be outdated or just plain WRONG

                                                                                                                                                                This is certainly an accurate statement, as evidenced by section 8.3.2 of the FQA. This was one of the places where I’d written notes for the FQA. Feel free to share any insight into Acme that cyclogram image gives you.

                                                                                                                                                                1. 2

                                                                                                                                                                  Yes, the discussion where the maintainer got upset opened with “I am now used to the FQA…” I’m not sure why you’re being pedantic here. The article also opens with “ The FQA is more harmful than good.”, two dismissals. And you did poke and prod: I would say asking two separate times about who created and/or committed the controversial image would be somewhat hair-raising.

                                                                                                                                                                  1. 2

                                                                                                                                                                    Yes, the discussion where the maintainer got upset opened with

                                                                                                                                                                    The discussion opened with the “What in the actual fuck” post, and KHM replied to dogstar’s post on the thread, not to mine.

                                                                                                                                                                    The article also opens with “ The FQA is more harmful than good.

                                                                                                                                                                    That statement is over 400 words into the piece. It is not the opening. Stop with the obvious falsehoods.

                                                                                                                                                                    And again, you’ve been unable to link to the poking and prodding because I already linked to the question I asked about who, which contains the reasoning for the question in the toot. Given you claim to be “truly a dispassionate observer here” it is clear you have an axe to grind. I will not discuss this with you further here as there is nothing either of us can say that would add to the conversation.

                                                                                                                                                            2. 3

                                                                                                                                                              how the author of this article felt the need to include 2 paragraphs about how the 9front devs are awful, terrible people for not immediately doing exactly what he asked.

                                                                                                                                                              1. Please tell me where in the post I said the devs are awful, terrible people.
                                                                                                                                                              2. Please show me a link where I’ve asked for the devs to take an immediate specific course of action.

                                                                                                                                                              You can’t because I did neither of those things. In fact it specifically says in the article that “9Front doesn’t owe me change”. If you’re going to say things that are untrue it helps if it’s not immediately verifiable. I’ve flagged this because it’s both untrue and doesn’t add to the discussion.

                                                                                                                                                              1. 2
                                                                                                                                                                1. “… and those feeling welcomed by it. My concerns lay with the 3rd group. The Nazi death camp joke author was so courteous…”

                                                                                                                                                                Emphasis mine. Come on man, are you really trying to argue that this is neutral? Where I come from, insinuating that a dev team are nazis is definitely saying that they’re bad people.

                                                                                                                                                                1. “ Update: The “9Front Internet Mob Mollification committee” huffed, puffed, some of them called me bad names then took it down.”

                                                                                                                                                                so this was your goal? What was your goal otherwise? In the mastodon thread, why were you trying to hold your (much desired, I’m sure) FQA additions over their heads?

                                                                                                                                                                If you don’t owe 9front any change, why mention them at all unless you want to punish them for some indiscretion? It doesn’t change the article in any way.

                                                                                                                                                                1. 3

                                                                                                                                                                  You keep cutting parts of things out, so I’ll put this here once.

                                                                                                                                                                  There are 3 types of people who’ll see it - Those who won’t use 9Front because of it, those whose choice is unaffected by it, and those feeling welcomed by it. My concerns lay with the 3rd group.

                                                                                                                                                                  I’m very clearly talking about people who would’ve seen the picture. Not 9Front Devs. People who look at stuff like that and think “Yeah, this is for me”.

                                                                                                                                                                  The Nazi death camp joke author was so courteous

                                                                                                                                                                  Look at the actual discussion thread and you will see how courteous the author was. I stand by my words, I did not say the devs were awful, terrible people as you falsely claimed.

                                                                                                                                                                  Update: The “9Front Internet Mob Mollification committee” huffed, puffed, some of them called me bad names then took it down.

                                                                                                                                                                  As it says, this is in a post-publish update, not the piece when you originally claimed. Just so we’re clear. The “9Front Internet Mob Moillification Committee” is a direct quote from the FQA text in Appendix L as represented in Mercurial.

                                                                                                                                                                  Huffed and puffed, some of them called me bad names - you can see all this here as well as in the mastodon thread.

                                                                                                                                                                  You’ve claimed to be truly a dispassionate observer. That claim is demonstrably false. You haven’t been able to link to a request for change because I never made one. You haven’t been able to show me calling the devs “awful, terrible people” because I never said that. Given that you’re obviously trolling I see no reason to engage further here, it won’t add anything to the overall discussion.

                                                                                                                                                                  1. 2

                                                                                                                                                                    Huffed and puffed, some of them called me bad names - you can see all this here as well as in the mastodon thread.

                                                                                                                                                                    There’s a saying: “If there are 10 normal people and 1 Nazi sitting at a table, there are 11 Nazis sitting at a table”. I know you’re aware of it, not only because it’s widely used, but because it was brought up in one of the threads you linked. You dismissed it. You were wrong to dismiss it.

                                                                                                                                                                    So, your insinuations, amplification of links to accusations, while insisting /your’e/ not saying anything is simply dancing around the edges of saying some very serious things about me and my friends, while not taking any responsibility.

                                                                                                                                                                    And, even worse: I believe on Mastodon, you implied that there was specific information you got via private messages. If this is true, and you did not speak out, then you’re also depriving me and my friends of tools cut these supposed shitbags off.

                                                                                                                                                                    So, either you’re helping spread baseless rumors about me and the company I keep, or you’re leaving me an accessory to the ideology that lead to the cold blooded murder of my family. I’m fairly certain it’s the former.

                                                                                                                                                                    And you’re wondering why I’m a little unhappy. Are you serious?

                                                                                                                                                                    It’s strange. It’s like you don’t actually grasp what’s being said – just words.

                                                                                                                                                                    Maybe it will click now.

                                                                                                                                                                    Or, if you ever end up in Jerusalem, maybe you can go to Yad Vashem. Maybe that will make it click. At least you could get a sensible, polite chuckle out of the make of the elevator they use to get in to the compound: Schindler’s Lifts.

                                                                                                                                                                    (And yes. There’s the question about gallows humor and why I accept, and even enjoy it. There’s a discussion to be had, and even academic papers to be read, about how it undermines oppression, both as an attack and as a defense, but I’m not keen to have it. Let’s leave it explained as me being an immature shithead with no taste. It saves breath, and I’m happy with that epiphet.. I will, however, point out that you can find (IMO) spicier Nazi jokes on fairly mainstream Israeli TV.)

                                                                                                                                                                    1. 4

                                                                                                                                                                      You’ve come into a thread where I wasn’t talking to you after you called me all kinds of names and swore at me on HN.

                                                                                                                                                                      know you’re aware of it, not only because it’s widely used, but because it was brought up in one of the threads you linked. You dismissed it. You were wrong to dismiss it.

                                                                                                                                                                      So you’re going through my Mastodon posts as well looking to cherry pick. You know full well I said:

                                                                                                                                                                      I’m not sure I agree with that last point. There is definitely a problem, but I think/hope the problem is more one of maturity than wolves in the flock.

                                                                                                                                                                      In a thread where the commentator thanked me for “calling the 9front people out on their nazi bullshit”. If I hadn’t responded I’m sure you’d be here now saying my lack of response would imply I agreed with their statement.

                                                                                                                                                                      You don’t know me. Our only interactions have involved you swearing at me, calling me names or accusing me. Please look at your own actions and stop harassing me. I will not respond to more of your harassment.

                                                                                                                                                          2. 5

                                                                                                                                                            I think kjs3 is trying to say that putting up a picture of Auschwitz is so unbelievably tasteless to a lot of people, that there should not be a discussion as to why it should be removed. This place of unbelievable crimes should never be used as an edgy joke. We all must be better than that.

                                                                                                                                                      2. 9

                                                                                                                                                        With context, it does sound kind of funny, and it might be the sort of thing I would have shared privately (with context) to friends or some such. But putting it up without context on a public FAQ is a pretty big failure in understanding that without this context, it just looks weird and out of place, at best.

                                                                                                                                                        Something like “heil Hitler” can be a literal endorsement of Hitler, but it can also be a joke (possibly in bad taste, but a joke nonetheless), or a statement against authoritarianism. Context is everything, and I don’t know why there’s such a failure to understand this from the 9front people. I suppose some of the more outlandish “oh, they must be literal Nazis then!” probably doesn’t help.

                                                                                                                                                        1. 3

                                                                                                                                                          I don’t understand what context would make someone think that image was anything other than an excessive attack on Ruby on Rails.

                                                                                                                                                          Do you know anyone that looks at that image and thinks “I love Rails, and therefore the juxtaposition must therefore that extermination camps are good!”?

                                                                                                                                                        2. 5

                                                                                                                                                          I take these accusations with a grain of salt, because they seem superficial. I’m yet to hear someone call them communists because they have the manifesto of the communist party in their repo. To me, it always was like a subversion of “”“optics”””, explicitly aimed at people who take one look and come to conclusions like OP, but I guess, when it doubt, people are Nazis (also known as not-“a political movement situated in the specific context of post-WW1 Germany”).

                                                                                                                                                          1. 4

                                                                                                                                                            The text of the Communist Manifesto replaced the text of Mein Kampf that was in the repo earlier.

                                                                                                                                                            It’s easiest if everyone just assumes the 9Front developers are 4chan-inflected trolls. It doesn’t really detract from the quality of the code, but it probably makes people considering contributing pause.

                                                                                                                                                        1. 2

                                                                                                                                                          Have you considered comparing this against other compression means that can use processors in parallel?

                                                                                                                                                          For example, tar can use pigz or other compression tools that can saturate all of te CPUs on your system.

                                                                                                                                                          1. 1

                                                                                                                                                            I had not, I will definitely consider adding it if there is a part 2 post.

                                                                                                                                                          1. 1

                                                                                                                                                            The single biggest issue I have with borg is the lack of support for using s3 buckets as a backup target. If bupstash supported this use case, I would happily switch to it regardless of how it performed performance-wise against borg.

                                                                                                                                                            1. 2

                                                                                                                                                              This is something I want to support, though it still requires a server side ‘gateway’ process and repository, but the bulk of the data is in external storage.

                                                                                                                                                            1. 3

                                                                                                                                                              I always come back to an encrypted volume and rsync.

                                                                                                                                                              /usr/local/bin/rsync -axHAX --partial --info=progress2 \
                                                                                                                                                                  --link-dest=../$OLD/ . /Volumes/backupvolA/$NEW/
                                                                                                                                                              

                                                                                                                                                              The nice thing about this setup is that all the files are sitting bare on a volume with each snapshot standing on its own as a collection of hardlinks and it works locally or over ssh.

                                                                                                                                                              Because it relies on rsync to do the delta, rsync has to enumerate both the source and the destination file systems and generate a high number of hardlinks for each new destination.

                                                                                                                                                              Would it be possible to add this to the comparison?

                                                                                                                                                              1. 1

                                                                                                                                                                I don’t really count this as the same thing because the server side needs access to the decryption key for this to work. I should perhaps have clarified that further.

                                                                                                                                                                1. 1

                                                                                                                                                                  Not necessarily – something like encfs/ecryptfs or a loop-mounted LUKS volume with its backend storage in a remote filesystem could keep the encryption local (this is actually what I do for my own backups using rsnapshot).

                                                                                                                                                                  1. 1

                                                                                                                                                                    That’s a pretty good idea, i think a remote loop file should perform well.

                                                                                                                                                              1. 2

                                                                                                                                                                Why didn‘t you benchmark duplicity? It is probably a lot slower, but it would be interesting how much slower.

                                                                                                                                                                Also you said, you want to push the performance further. I would recommend you to improve the software where it is most needed. Your software is already best in class performance-wise and users were probably fine with the performance of the other solutions beforehand. It feels great to improve performance as it is measured so easily and clearly.

                                                                                                                                                                1. 2

                                                                                                                                                                  Why didn‘t you benchmark duplicity?

                                                                                                                                                                  Only because I have never used that before, It might be worth trying for a part 2 post.

                                                                                                                                                                  I would recommend you to improve the software where it is most needed.

                                                                                                                                                                  Bupstash makes other improvements to areas I found lacking, mainly around access controls and offline decryption keys. I do have plans for other improvements too, its just this post was focused on performance. Part of the motivation of this post was to attract new users for feedback, a chicken and egg problem.

                                                                                                                                                                  1. 1

                                                                                                                                                                    Thanks for the fast response. The improvements bupstash brings to the table are great and because of that I’m just about make the first backup with it along duplicity. I was just saying don’t focus on performance too much :D

                                                                                                                                                                    Thank you for you effort!

                                                                                                                                                                1. 2

                                                                                                                                                                  I’m currently using restic to backup to Backblaze B2. I’m actually pretty keen to switch to bupstash given it seems quite a bit faster. Are there any plans to support object stores like B2 or S3 or is the requirement for a server process likely to remain an obstacle there?

                                                                                                                                                                  1. 3

                                                                                                                                                                    Bupstash supports external storage via a plugin interface, this is evolving but definitely something that is in the works.

                                                                                                                                                                    1. 1

                                                                                                                                                                      Excellent. I’ll continue to keep an eye on the progress.

                                                                                                                                                                  1. 11

                                                                                                                                                                    I wonder how this would compare with zfs send and encrypted snapshots.

                                                                                                                                                                    1. 6

                                                                                                                                                                      I’d also like to see zstd swapped in for gzip.

                                                                                                                                                                      1. 2

                                                                                                                                                                        I’m not convinced the performance tradeoffs make ZFS’ zstd implementation a worthy replacement for the good speed/performance balance of lz4, FWIW. Definitely a replacement for gzip though.

                                                                                                                                                                      2. 2

                                                                                                                                                                        My guess is using raw sends would be better in almost all categories except maybe compression.

                                                                                                                                                                        1. 6

                                                                                                                                                                          This reminds me of an interview questions I heard about a while back.

                                                                                                                                                                          Is it better to encrypt then compress or compress then encrypt?

                                                                                                                                                                          The incorrect “correct” answer is that is to compress then encrypt, as a encrypted blob should be relatively indistinguishable from random noise so compressing an encrypted blob will do very little. So you should always compress and encrypt.

                                                                                                                                                                          The correct response is that compressing and encrypt can leak information about the data being encrypted. There is the BEAST & CRIME as to obvious examples of this, but in the example of an compress and encrypted audio stream you can actually use variation in packet size to approximate what is being said.

                                                                                                                                                                          So the answer is, as always, it depends, but generally for security just encrypt it and send it. Compressing it is largely useless, or potentially harmful.

                                                                                                                                                                          1. 3

                                                                                                                                                                            I think that’s more for client-server protocols, though? In practice, the threat model for ZFS encryption is someone with physical access to your disk or someone who can intercept the send stream. The manpage warns about using encryption and compression (as well as encryption and deduplication, but that one is obvious to me, dedup weakens the encryption in a really obvious way) - but I’m struggling to see how someone could mount a CRIME attack against the same disk. Maybe multitenant environments?

                                                                                                                                                                            Since compression is applied before encryption datasets may be vulnerable to a CRIME-like attack if applications accessing the data allow for it. Deduplication with encryption will leak information about which blocks are equivalent in a dataset and will incur an extra CPU cost per block written.

                                                                                                                                                                            1. 4

                                                                                                                                                                              My understanding is a lot of these attacks depend on the attacker being able to induce the client to encrypt attacker controlled data and also be able to see the encrypted result. This seems extremely difficult (though not totally impossible) with tools such as bupstash, this is because backups are largely one way transmissions and offline interactions often at some point far in the future.

                                                                                                                                                                        2. 2

                                                                                                                                                                          I also experimented with btrfs send into a bupstash repository, which seems to work quite well, though is not totally incremental like what you suggest. The bupstash send log at least cuts down on network traffic, though in that case it still requires we read the whole snapshot from disk.

                                                                                                                                                                          https://bupstash.io/doc/guides/Filesystem%20Backups.html#Btrfs-send-snapshots.

                                                                                                                                                                        1. 2

                                                                                                                                                                          Could it be used on Windows?

                                                                                                                                                                          1. 4

                                                                                                                                                                            I haven’t tested bupstash on windows yet, but it’s something I plan to make work, I suspect it might need some fixes first,

                                                                                                                                                                            1. 1

                                                                                                                                                                              It also means you have to implement VSS support into bupstash, because backups on windows without supporting the VSS features wont make any sense ..