1. 2

    I just switched to OpenBSD for e-mail using the following stack:

    Inbound: opensmtpd -> spampd(tag) -> opensmtpd -> clamsmtpd(tag) -> opensmtpd -> procmail -> dovecot(Maildir) outbound: opensmtpd -> dkim_proxy -> opensmtpd(relay)

    I don’t use the spamd/grey listing up front like a lot of tutorials suggest, but spampd(spam assistant) seems to get the majority of it.

    My old stack was similar, but used postfix on opensuse. I really like the opensmtpd configuration; loads simpler than postfix. However I wish it supporter filters that the other MTAs do. It had filter support for a bit, but was clunky and subsequently removed. It makes it difficult (impossible?) to run things like rspam.

    1. 5

      rspamd has an MDA mode, so you can do like

      accept from any for local virtual { "@" => mike } deliver to mda "rspamc --mime --ucl --exec /usr/loca
      l/bin/dovecot-lda-mike" as mike
      

      and dovecot-lda-mike is

      #! /bin/sh
      exec /usr/local/libexec/dovecot/dovecot-lda -d mike
      

      smtpd is really really really good. For some reason the email software ecosystem is a mess of insane configs and horrible scripts, but my smtpd.conf is 12 lines and the only script I use (that rspamd one) is going to go away when filters come back. smtpd is so good I went with an MDA instead of a web app to handle photo uploads to my VPS. It’s one line in smtpd.conf and ~70 lines of python, and I don’t have to deal with fcgi or anything like that.

      1. 1

        smtpd is so good I went with an MDA instead of a web app to handle photo uploads to my VPS

        Oh that’s a clever idea. I’ve been using ssh (via termux) on my phone but that is so clumsy.

      2. 5

        I do greylisting on my email server [1] and I’ve found that it reduces the incoming email by 50% up front—there are a lot of poorly written spam bots out there. Greylisting up front will reduce the load that your spam system will have to slog through, for very little cost.

        [1] Yes, I run my own. Been doing it nearly 20 years now (well over 10 at its current location) so I have it easier than someone starting out now. Clean IP, full control over DNS (I run my own DNS server; I also have access to modify the PTR record if I need to) and it’s just me—no one else receives email from my server.

        1. 2

          I’m the author/presenter of the tutorial. If I may, I suggest looking at my talk this year at BSDCan: Fighting Spam at the Frontline: Using DNS, Log Files and Other Tools in the Fight Against Spam. In those slides I talk about using spf records (spf_fetch, smtpctl spfwalk, spfwalk standalone) to whitelist IPs and mining httpd and sshd logs for bad actors and actively blacklisting them.

          For those who find blacklisting a terrifying idea, in the presentation I suggest configuring your firewall rules so that your whitelists always win. That way, if Google somehow get added to your blacklists, the whitelist rule will ensure Gmail can still connect.

          I also discuss ways to capture send-to domains and add them to your whitelists so you don’t have to wait hours for them to escape the greylists.

          1. 1

            I didn’t find SPF to be all that great, and it was the nearly the same three years earlier. Even the RBL were problematic, but that was three years ago.

            As for greylisting, I currently hold them for 25 minutes, and that might be 20 minutes longer than absolutely required.

          2. 1

            Greylisting is the best. Back when my mailserver was just on a VPS it was the difference between spamd eating 100% CPU and a usable system.

        1. 12

          As someone who uses arch on all my developer machines, arch is a horrible developer OS, and I only use it because I know it better than other distros.

          It was good 5-10 years ago (or I was just less sensitive back then), but now pacman Syu is almost guaranteed to break or change something for the worse, so I never update, which means I can never install any new software because everything is dynamically linked against the newest library versions. And since the arch way is to be bleeding edge all the time, asking things like “is there an easy way to roll back an update because it broke a bunch of stuff and brought no improvements” gets you laughed out the door.

          I’m actually finding myself using windows more now, because I can easily update individual pieces of software without risking anything else breaking.

          @Nix people: does NixOS solve this? I believe it does but I haven’t had a good look at it yet.

          1. 14

            Yes, Nix solves the “rollback” problem, and it does it for your entire OS not just packages installed (config files and all).

            With Nix you can also have different versions of tools installed at the same time without the standard python3.6 python2.7 binary name thing most place do: just drop into an new nix-shell and install the one you want and in that shell that’s what you have. There is so much more. I use FreeBSD now because I just like it more in total, but I really miss Nix.

            EDIT: Note, FreeBSD solves the rollback problem as well, just differently. In FreeBSD if you’re using ZFS, just create a boot environment before the upgrade and if the upgrade fails, rollback to the pre-upgrade boot environment.

            1. 9

              Being a biased Arch Developer, I rarely have Arch break when updating. Sometimes I have to recompile our own C++ stack due to soname bumps but for the rest it’s stable for me.

              For Arch there is indeed no rollback mechanism, although we do provide an archive repository with old versions of packages. Another option would be BTRFS/ZFS snapshots. I believe the general Arch opinion is instead of rolling back fixing the actual issue at hand is more important.

              1. 8

                I believe the general Arch opinion is instead of rolling back fixing the actual issue at hand is more important.

                I can see some people might value that perspective. For me, I like the ability to plan when I will solve a problem. For example I upgraded to the latest CURRENT in FreeBSD the other day and it broke. But I was about to start my work day so I just rolled back and I’ll figure out when I have time to address it. As all things, depends on one’s personality what they prefer to do.

                1. 2

                  For me, I like the ability to plan when I will solve a problem.

                  But on stable distros you don’t even have that choice. Ubuntu 16.04, (and 18.04 as well I believe) ships an ncurses version that only supports up to 3 mouse buttons for ABI stability or something. So now if I want to use the scroll wheel up, I have to rebuild everything myself and maintain some makeshift local software repository.

                  And that’s not an isolated case, from a quick glance at my $dayjob workstation, I’ve had to build locally the following: cquery, gdb, ncurses, kakoune, ninja, git, clang and other various utilities. Just because the packaged versions are ancient and missing useful features.

                  On the other hand, I’ve never had to do any of this on my arch box because the packaged software is much closer to upstream. And if an update break things, I can also roll back from that update until I have time to fix things.

                  1. 2

                    I don’t use Ubuntu and I try to avoid Linux, in general. I’m certainly not saying one should use Ubuntu.

                    And if an update break things, I can also roll back from that update until I have time to fix things.

                    Several people here said that Arch doesn’t really support rollback which is what I was responding to. If it supports rollback, great. That means you can choose when to solve a problem.

                    1. 1

                      I don’t use Ubuntu and I try to avoid Linux, in general. I’m certainly not saying one should use Ubuntu.

                      Ok, but that’s a problem inherent to stable distros, and it gets worse the more stable they are.

                      Several people here said that Arch doesn’t really support rollback

                      It does, pacman keeps local copies of previous versions for each package installed. If things break, you can look at the log and just let pacman install the local package.

                      1. 1

                        It does, pacman keeps local copies of previous versions for each package installed. If things break, you can look at the log and just let pacman install the local package.

                        Your description makes it sound like pacman doesn’t support roll backs, but you can get that behaviour if you have to and are clever enough. Those seem like very different things to me.

                        Also, what you said about stable distros doesn’t seem to match my experience in FreeBSD. FreeBSD is ‘stable’ however ports packages tend to be fairly up to date (or at least I rarely run into it except for a few).

                        1. 1

                          I’m almost certain any kind of “rollback” functionality in pacman is going to be less powerful than what’s in Nix, but it is very simple to rollback packages. An example transcript:

                          $ sudo pacman -Syu
                          ... some time passes, after a reboot perhaps, and PostgreSQL doesn't start
                          ... oops, I didn't notice that PostgreSQL got a major version bump, I don't want to deal with that right now.
                          $ ls /var/cache/pacman/pkg | rg postgres
                          ... ah, postgresql-x.(y-1) is sitting right there
                          $ sudo pacman -U /var/cache/pacman/pkg/postgres-x.(y-1)-x86_64.pkg.tar.xz
                          $ sudo systemctl start postgres
                          ... it's alive!
                          

                          This is all super standard, and it’s something you learn pretty quickly, and it’s documented in the wiki: https://wiki.archlinux.org/index.php/Downgrading_packages

                          My guess is that this is “just downgrading packages” where as “rollback” probably implies something more powerful. e.g., “rollback my system to exactly how it was before I ran the last pacman -Syu.” AFAIK, pacman does not support that, and it would be pretty tedious to actually do it if one wanted to, but it seems scriptable in limited circumstances. I’ve never wanted/needed to do that though.

                          (Take my claims with a grain of salt. I am a mere pacman user, not an expert.)

                          EDIT: Hah. That wiki page describes exactly how to do rollbacks based on date. Doesn’t seem too bad to me at all, but I didn’t know about it: https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_restore_all_packages_to_a_specific_date

            2. 12

              now pacman Syu is almost guaranteed to break or change something for the worse

              I have the opposite experience. Arch user since 2006, and updates were a bit more tricky back then, they broke stuff from time to time. Now nothing ever breaks (I run Arch on three different desktop machines and two servers, plus a bunch of VMs).

              I like the idea of NixOS and I have used Nix for specific software, but I have never made the jump because, well, Arch works. Also with Linux, package management has never been the worst problem, hardware support is, and the Arch guys have become pretty good at it.

              1. 3

                I have the opposite experience

                I wonder if the difference in experience is some behaviour you’ve picked up that others haven’t. For example, I’ve found that friend’s children end up breaking things in ways that I would never do just because I know enough about computers to never even try it.

                1. 2

                  I think it’s a matter of performing Syu update often (every few days or even daily) instead of once per month. Rare updates indeed sometimes break things but when done often, it’s pretty much update and that’s it.

                  I’m an Arch user since 6 years and there were maybe 3 times during those 6 years where something broke badly (I was unable to boot). Once it was my fault; second & third one is related to nvidia driver and Xorg incompatibility.

                  1. 3

                    Rare updates indeed sometimes break things but when done often, it’s pretty much update and that’s it.

                    It’s sometimes also a matter of bad timing. Now every time before doing a pacman -Syu I check /r/archlinux and the forums to see if someone is complaining. If so then I tend to wait for a day or two before the devs push out updates to broken packages.

                  2. 1

                    That’s entirely possible.

                2. 4

                  I have quite a contrary experience, I have pacman run automated in the background every 60 minutes and all breakage I suffer is from human-induced configuration errors (such as misconfigured boot loader or fstab)

                  1. 1

                    Things like Nix even allow rolling back from almost all user configuration errors.

                    1. 3

                      Would be nice, yeah, though I never understood or got Nix really. It’s a bit complicated and daunting to get started and I found the documentation to be lacking.

                  2. 3

                    How often were you updating? Arch tends to work best when it’s updated often. I update daily and can’t remember the last time I had something break. If you’re using Windows, and coming back to Arch very occasionally and trying to do a huge update you may run into conflicts, but that’s just because Arch is meant to be kept rolling along.

                    I find Arch to be a fantastic developer system. It lets me have access to all the tools I need, and allows me to keep up the latest technology. It also has the bonus of helping me understand what my system is doing, since I have configured everything.

                    As for rollbacks, I use ZFS boot environments. I create one prior to every significant change such as a kernel upgrade, and that way if something did happen go wrong, and it wasn’t convenient to fix the problem right away, I know that I can always move back into the last environment and everything will be working.

                    1. 2

                      How do you configure ZFS boot environments with Arch? Or do you just mean snapshots?

                      1. 3

                        I wrote a boot environment manager zedenv. It functions similarly to beadm. You can install it from the AUR as zedenv or zedenv-git.

                        It integrates with a bootloader if it has a “plugin” to create boot entries, and keep multiple kernels at the same time. Right now there’s a plugin for systemdboot, and one is in the works for grub, it just needs some testing.

                        1. 2

                          Looks really useful. Might contribute a plugin for rEFInd at some point :-)

                          1. 1

                            Awesome! If you do, let me know if you need any help getting started, or if you have any feedback.

                            It can be used as is with any bootloader, it just means you’ll have to write the boot config by hand.

                  1. 1

                    I like Understanding Digital Signal Processing but it doesn’t say anything about compression. I have Introduction to Data Compression which does cover audio compression but I haven’t read that far yet, and the bits I have read have been incredibly dry material and not super useful so I don’t recommend it.

                    Not sure if it helps but a couple of my “aha” moments learning signal processing:

                    If you know projecting a vector onto a set of basis vectors from linear algebra, the DFT is projecting your signal onto a set of vectors with sin waves of different frequencies in them. Coming from a compsci background that was the easiest explanation for me to understand.

                    A lowpass filter is like multiplying the frequency domain version of your signal by [1, 1, 1, ..., 0, 0, 0], where the change from 1 to 0 happens at the cutoff frequency. Multiplication in the frequency domain is equivalent to convolution in the time domain, so sure enough you can take the IDFT of [1, 1, 1, ..., 0, 0, 0] and convolve that with things to get a (crappy) lowpass filter.

                    1. 3

                      This is silly drama about something we already have far too many submissions for.

                      1. 23

                        This is a bit disappointing. It feels a bit like we are walking into the situation OpenGL was built to avoid.

                        1. 7

                          To be honest we are already in that situation.

                          You can’t really use GL on mac, it’s been stuck at D3D10 feature level for years and runs 2-3x slower than the same code under Linux on the same hardware.

                          It always seemed like a weird decision from Apple to have terrible GL support, like if I was going to write a second render backend I’d probably pick DX over Metal.

                          1. 6

                            I remain convinced that nobody really uses a Mac on macOS for anything serious.

                            And why pick DX over Metal when you can pick Vulkan over Metal?

                            1. 3

                              Virtually no gaming or VR is done on a mac. I assume the only devs to use Metal would be making video editors.

                              1. 1

                                This is a bit pedantic, but I play a lot of games on mac (mainly indie stuff built in Unity, since the “porting” is relatively easy), and several coworkers are also mac-only (or mac + console).

                                Granted, none of us are very interested in the AAA stuff, except a couple of games. But there’s definitely a (granted, small) market for this stuff. Luckily stuff like Unity means that even if the game only sells like 1k copies it’ll still be a good amount of money for “provide one extra binary from the engine exporter.”

                                The biggest issue is that Mac hardware isn’t shipping with anything powerful enough to run most games properly, even when you’re willing to spend a huge amount of money. So games like Hitman got ported but you can only run it on the most expensive MBPs or iMac Pros. Meanwhile you have sub-$1k windows laptops which can run the game (albeit not super well)

                              2. 2

                                I think Vulkan might have not been ready when Metal was first skecthed out – and Apple does not usually like to compromise on technology ;)

                                1. 2

                                  My recollection is that Metal appeared first (about June 2014), Mantle shipped shortly after (by a coupe months?), DX12 shows up mid-2015 and then Vulkan shows up in February 2016.

                                  I get a vague impression that Mantle never made tremendous headway (because who wants to rewrite their renderer for a super fast graphics API that only works on the less popular GPU?) and DX12 seems to have made surprisingly little (because targeting an API that doesn’t work on Win7 probably doesn’t seem like a great investment right now, I guess? Current Steam survey shows Win10 at ~56% and Win7+8 at about 40% market share among people playing videogames.)

                                  1. 2

                                    Mantle got heavily retooled into Vulkan, IIRC.

                                    1. 1

                                      And there was much rejoicing. ♥

                          1. 4

                            This whole area has been exercising my brain recently.

                            As much as I hate the C standard committee’s lack of courage in defining behaviour, as often a simple decision, even if controversial will resolve it.

                            However, here is one that is sort of unresolvable.

                            What behaviour should a program have that indexes beyond the bounds of an array?

                            There is no way a standard can prescribe what the result will be.

                            It must be undefined.

                            So the obvious thing to do, as Pascal did, is do bounds checking and have a defined behaviour.

                            That imposes substantial runtime costs in CPU and memory, so users do switch it off…..

                            So what should the behaviour be?

                            One reasonable assumption a compiler writer can make is that there is no way the programmer can intend to index out of bounds, so I can assume that the index is less than the bound and generate machine code accordingly.

                            You might say, these newfangled optimizations are the problem… no they aren’t.

                            Compilers have been relaying out data in memory according what they think best for decades.

                            Where this whole thing is driving me nuts is around asserts. See this thread I started here… https://readlist.com/lists/gcc.gnu.org/gcc-help/7/39051.html

                            Asserts, if they are compiled in, tell the compiler (without getting involved in UB optimizations) that if expression is false, then everything down stream is not reachable…. so it analyzes under the assumption the expression is true.

                            However it doesn’t attempt to warn you if it finds a code path where the expression is false, and completely redoes it’s optimization without that assumption if you compile the assert out.

                            1. 4

                              What behaviour should a program have that indexes beyond the bounds of an array?

                              There is no way a standard can prescribe what the result will be.

                              It must be undefined.

                              This gets to the heart of the matter, I think. Part of the issue is people confuse “the language standard” with “what compilers do”. The language says it is undefined behaviour for an out-of-bounds array access to occur, or for signed integers to have their value range exceeded, but there’s no reason why compilers can’t generate code which will explicitly detect these situations and throw out an error message (and terminate).

                              So why don’t compilers generally do that by default? Because C is used in performance critical code where these checks have a cost which is considered significant. And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

                              If you do value deterministic behaviour on program error and are willing to sacrifice some performance to get it, the obvious solution is to use a language which provides that, i.e. don’t use C. But that’s not a justification to criticise the whole concept of undefined behaviour in C.

                              1. 4

                                There is a false choice between inefficient code with run time bounds checking and compiler “optimizations” that break working code. I love the example in http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_2015_submission_29.pdf where the GCC developers introduce a really stupid UB based “optimization” that broke working code and then found, to their horror, that it broke a benchmark. So they disabled it for the benchmark.

                                And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

                                Great. Let’s see an example.

                                But that’s not a justification to criticise the whole concept of undefined behaviour in C.

                                I think this attitude comes from a fundamental antipathy to the design of C or a basic misunderstanding of how it is used. C is not Java or Swift - and not because its designers were stupid or mired in archaic technology.

                                1. 4

                                  There is a false choice between inefficient code with run time bounds checking and compiler “optimizations” that break working code

                                  Optimisations don’t break working code. They cause broken code to have different observable behaviour.

                                  And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

                                  Great. Let’s see an example.

                                  I don’t have a code example to hand, and as I said they’re not trivial, but that doesn’t mean it’s not true. Since it can eliminate whole code paths, it can affect the efficacy of for example value range propagation, affect inlining decisions, and have other flow-on effects.

                                  I think this attitude comes from a fundamental antipathy to the design of C or a basic misunderstanding of how it is used

                                  I disagree.

                                  1. -1

                                    Optimisations don’t break working code. They cause broken code to have different observable behaviour.

                                    That’s a legalistic answer. The code worked as expected and produced the correct result. The “optimization” caused it to misfunction.

                                    I don’t have a code example to hand

                                    Apparently nobody does. So the claimed benefit is just hand waving.

                                    I disagree.

                                    The thinking of Lattner is indicative. He agrees that compiler behavior using the UB loophole makes C a minefield. His solution is to advocate Swift. People who are hostile to the use of C should not be making these decisions.

                                    1. 5

                                      That’s a legalistic answer.

                                      Alas, in absence of “legalistic answers”, the only definition of C is either…

                                      • An implementation of C is a valid implementation iff every program in a Blessed Set of programs compile and runs successfully and outputs exactly the same values.

                                        or

                                      • An implementation of C is a valid implementation iff, every program that compiles and runs successfully on The One True Blessed C Compiler, compiles and runs and outputs exactly the same values AND every program that fails to compile on The One True Blessed Compiler, fails to compile on the candidate compiler.

                                      What sort of C are you envisioning?

                                      Those may be appropriate ways to progress, but that is a different language and probably should be called something other than C.

                                      1. 3

                                        Apparently nobody does. So the claimed benefit is just hand waving

                                        Again, I don’t agree.

                                        1. 1

                                          You can disagree all you want, but you also seem to be unable to produce any evidence.

                                          1. 3

                                            You can disagree all you want, but you also seem to be unable to produce any evidence.

                                            I have high confidence that I could produce, given some time, an example of code which compiled to say 20 instructions if integer overflow were defined and just 1 or 2 otherwise, and probably more by abusing the same technique repeatedly, but you might then claim it wasn’t representative of “real code”. And then if I really wanted to satisfy you I would have to find some way to trawl through repositories to identify some piece of code that exhibited similar properties. It’s more work than I care to undertake to prove my point here, and so I suppose you have a right to remain skeptical.

                                            On the other hand, I have at least explained (even if only very briefly) how small optimisations such as assuming that integer arithmetic operations won’t overflow could lead to significant differences in code generation, beyond simple exchanging of instructions. You’ve given no argument as to why this couldn’t be the case. So, I don’t think there’s any clearly stronger argument on either side.

                                            1. 0

                                              I have high confidence that I could produce, given some time, an example of code which compiled to say 20 instructions if integer overflow were defined and just 1 or 2 otherwise

                                              I have no confidence of this, and it would be a completely uninteresting optimization in any case.

                                              On the other hand, I have at least explained (even if only very briefly) how small optimisations such as assuming that integer arithmetic operations won’t overflow could lead to significant differences in code generation, beyond simple exchanging of instructions.

                                              Not really. You are omitting a single instruction that almost certainly costs no cycles at all in a modern pipelined processor. Balance that against putting minefields into the code - and note there is no way in C to check for this condition. The tradeoff is super unappealing.

                                              1. 2

                                                Not really. You are omitting a single instruction

                                                No, I was not talking about omitting a single instruction.

                                2. 2

                                  With assert(), you are telling the compiler that at this point, this is true. The compiler is trusting your assertion of the truth at that point.

                                  Also, if you compile with -DNDEBUG -O3 you will get the warning:

                                  [spc]saltmine:/tmp>gcc -std=c99 -Wall -Wextra -pedantic -DNDEBUG -O3 c.c
                                  c.c: In function ‘main’:
                                  c.c:7:20: warning: ‘*((void *)&a+10)’ is used uninitialized in this function [-Wuninitialized]
                                  c.c:13:8: note: ‘a’ was declared here
                                  [spc]saltmine:/tmp>gcc -std=c99 -Wall -Wextra -pedantic -O3 c.c
                                  [spc]saltmine:/tmp>
                                  
                                  1. 2

                                    No, that is a meaningless statement.

                                    The compiler doesn’t even see an assert statement, let alone “trust it”.

                                    It is a macro that gets expanded to “plain old code” at preprocessor time, so depending on NDEBUG settings it expands either to something like if(!(exp))abort() or nothing.

                                    What the compiler does trust is the __attribute_((__noreturn__)) on the abort() function.

                                    1. 1

                                      My file:

                                      #include <assert.h>
                                      
                                      int foo(int x)
                                      {
                                        assert(x >= 0);
                                        return x + 5;
                                      }
                                      

                                      My file after running it through the C preprocessor:

                                      # 1 "x.c"
                                      # 1 "/usr/include/assert.h"
                                       
                                       
                                      
                                       
                                       
                                       
                                      
                                       
                                      # 12
                                      
                                      # 15
                                      
                                      #ident      "@(#)assert.h   1.10    04/05/18 SMI"
                                      
                                      # 21
                                      
                                      # 26
                                      extern void __assert(const char *, const char *, int);
                                      # 31
                                      
                                      # 35
                                      
                                      # 37
                                      
                                       
                                      # 44
                                      
                                      # 46
                                      
                                      # 52
                                      
                                      # 63
                                      
                                      # 2 "x.c"
                                      
                                      int foo(int x)
                                      {
                                         ( void ) ( ( x >= 0 ) || ( __assert ( "x >= 0" , "x.c" , 5 ) , 0 ) );
                                        return x + 5;
                                      }
                                      #ident "acomp: Sun C 5.12 SunOS_sparc 2011/11/16"
                                      

                                      Not an __atttribute__ to be found. This C compiler can now generate code as if x is never a negative value.

                                      1. 1

                                        #include <assert.h> int foo(int x) { assert(x >= 0); return x + 5; }

                                        Can you copy paste the assembly output? (I read sparc asm as part of my day job….)

                                        I’d be interested to see is it is treating __assert() as anything other than a common or garden function.

                                        1. 1

                                          I’m not sure what it’ll prove, but okay:

                                              cmp     %i0,0
                                              bge     .L18
                                              nop
                                          
                                          .L19:
                                              sethi   %hi(.L20),%o0
                                              or      %o0,%lo(.L20),%o0
                                              add     %o0,8,%o1
                                              call    __assert
                                              mov     6,%o2
                                              ba      .L17
                                              nop
                                          
                                              ! block 3
                                          .L18:
                                              ba      .L22
                                              mov     1,%i5
                                          
                                              ! block 4
                                          .L17:
                                              mov     %g0,%i5
                                          
                                              ! block 5
                                          .L22:
                                          
                                          !    7    return x + 5;
                                          
                                              add     %i0,5,%l0
                                              st      %l0,[%fp-4]
                                              mov     %l0,%i0
                                              jmp     %i7+8
                                              restore
                                          
                                              ! block 6
                                          .L12:
                                              mov     %l0,%i0
                                              jmp     %i7+8
                                              restore
                                          

                                          I did not specify any optimizations, and from what I can tell, it calls a function called __assert().

                                          1. 1

                                            TL;DR; The optimiser for this compiler is crap. And it isn’t treating __assert() as special / noreturn.

                                            int foo(int x)
                                            {
                                               ( void ) ( ( x >= 0 ) || ( __assert ( "x >= 0" , "x.c" , 5 ) , 0 ) );
                                              return x + 5;
                                            }
                                            
                                                ;; x is register %i0
                                                cmp     %i0,0               ; Compare x with 0
                                                bge     .L18                ; If it is large branch to .L18
                                                nop                         ; Delay slot. Sigh sparc pipelining is makes debugging hard.
                                            
                                            ;;; This is the "call assert" branch. gcc has function __attribute__((cold)) or
                                            ;;; __builtin_expect() to mark this as the unlikely path.
                                            .L19:
                                                sethi   %hi(.L20),%o0       
                                                or      %o0,%lo(.L20),%o0
                                                add     %o0,8,%o1
                                                call    __assert
                                                mov     6,%o2               ;Delay slot again
                                                ba      .L17                ; Branch absolute to .L17
                                                nop                         ;Delay slot
                                            
                                                ;; Really? Is this optimized at all?
                                                ! block 3
                                            .L18:
                                                ba      .L22                ; Branch absolute to .L22!
                                                mov     1,%i5               ; put 1 in %i5
                                            
                                            ;;; Seriously? Is this thing trying to do it the hard way?
                                            ;;; The assert branch sets %i5 to zero.
                                                ! block 4
                                                .L17:
                                                ;; Fun fact. %g0 is the sparc "bit bucket" reads as zero, ignores anything written to it.
                                                mov     %g0,%i5             
                                            
                                                ! block 5
                                                ;; Falls through. ie. Expected to come 
                                                ;; out of __assert() *hasn't treated __assert as noreturn!*
                                            
                                                ;; Joins with the x>=0 branch
                                            .L22:
                                            
                                            !    7    return x + 5;
                                                ;; Local register %l0 is x + 5
                                                add     %i0,5,%l0
                                                st      %l0,[%fp-4]         ;WTF? Has this been inlined into a larger block of code?
                                                mov     %l0,%i0             ;WTF? as above?
                                                jmp     %i7+8               ;Return to calling addres.
                                                restore                     ;Unwind sparc register windowing.
                                            
                                                ;; WTF? No reference to label .L12
                                                ! block 6
                                            .L12:
                                                mov     %l0,%i0
                                                jmp     %i7+8
                                                restore
                                            
                                    2. 1

                                      Actually, that isn’t quite what happens….

                                      Actually, it’s “Just Another Macro” which, very approximately, expands to …

                                       if( !(exp)) abort();
                                      

                                      …where abort() is marked __attribute__((noreturn));

                                      Which is almost, but not quite what one would want….

                                      As the compiler uses the noreturn attribute to infer that if !exp, then rest of code is unreachable, therefore for rest of code exp is true.

                                      Alas, I have found that it doesn’t, if it finds a path for which exp is false, warn you that you will abort!

                                      I certainly feel there is room for compiler and optimizer writers work with design by contract style programmers to have a “mutually beneficial” two way conversation with the programmers when they write asserts.

                                      1. 0

                                        Which is almost, but not quite what one would want….

                                        I’m not sure I understand you. assert() will abort if the expression given is false. That’s what it does. It also prints where the expression was (it’s part of the standard). If you don’t want to abort, don’t call assert(). If you expect that assert() is a compile-time check, well, it’s not.

                                        I certainly feel there is room for compiler and optimizer writers work with design by contract style programmers to have a “mutually beneficial” two way conversation with the programmers when they write asserts.

                                        There’s only so far that can go though. Put your foo() function in another file, and no C compiler can warn you.

                                        assert() is also a standard C function, which means the compiler can have built-in knowledge of its semantics (much like a C compiler can replace a call to memmove() with inline assembly). The fact that GCC uses its __attribute__ extension for this doesn’t apply to all other compilers.

                                        1. 2

                                          That’s the other bit of Joy about C.

                                          There are two entirely separate things….

                                          The compiler.

                                          And the standard library.

                                          gcc works quite happily with several entirely different libc’s.

                                          assert.h is part of libc, not the compiler.

                                          How assert() is implemented is the domain of the libc implementer not the compiler writer.

                                          I have poked at quite a few different compilers and even more libc’s…. as I have summarised is how all I have looked at are doing things. (Although some don’t have a concept of “noreturn” so can’t optimize based on that)

                                          Which compiler / libc are you looking at?

                                          1. 3

                                            The compiler that comes with Solaris right now.

                                            You can’t have a standard C compiler without the standard C library. I can get a compiler that understands, say, C99 syntax, but unless it comes with the standard C library, it can’t be called a compliant C99 compiler. The standard covers both the language and the library. I’m reading the C99 standard right now, and here’s an interesting bit:

                                            Each library function is declared, with a type that includes a prototype, in a header, (182) whose contents are made available by the #include preprocessing directive.

                                            And footnote 182 states:

                                            (182) A header is not necessarily a source file, nor are the < and > delimited sequences in header names necessarily valid source file names.

                                            To me, that says the compiler can have knowledge of the standard functions. Furthermore:

                                            Any function declared in a header may be additionally implemented as a function-like macro defined in the header … Likewise, those function-like macros described in the following subclauses may be invoked in an expression anywhere a function with a compatible return type could be called.(187)

                                            (187) Because external identifiers and some macro names beginning with an underscore are reserved, implementations may provide special semantics for such names. For example, the identifier _BUILTIN_abs could be used to indicate generation of in-line code for the abs function. Thus, the appropriate header could specify #define abs(x) _BUILTIN_abs(x) for a compiler whose code generator will accept it. In this manner, a user desiring to guarantee that a given library function such as abs will be a genuine function may write #undef abs whether the implementation’s header provides a macro implementation of abs or a built-in implementation. The prototype for the function, which precedes and is hidden by any macro definition, is thereby revealed also.

                                            So the compiler can absolutely understand the semantics of standard C calls and treat them specially. Whether a C compiler does so is implementation defined. And good luck writing offsetof() of setjmp()/longjmp() portably (spoiler: you can’t—they’re tied to both the compiler and architecture).

                                            So, getting back to assert() and your issues with it. Like I said, the compilers knows (whether it’s via GCC’s __attribute__(__noreturn__) or because the compiler has built-in knowledge of the semantics of assert()) that the expression used must be true and can thus optimize based on that information, much like it can remove the if statement and related code:

                                            const int debug = 0;
                                            
                                            {
                                              int x = debug;
                                            
                                              if (x)
                                              {
                                                fprintf(stderr,"here we are!\n");
                                                exit(33);
                                              }
                                              // ...
                                            }
                                            

                                            even through x, because debug is constant, x is loaded with a constant, and not modified prior to the if statement. Your wanting a warning about an invalid index to an array whose index is used in assert() is laudable, but to the compiler, you are telling it “yes, this is fine. No complaints please.” Compile the same code with NDEBUG defined, the assert() goes away (from the point of view of the compiler phase) and the diagnostic can be issued.

                                            Yes, it sucks. But that’s the rational.

                                            The intent is you run the code, you get the assert, you fix the code (otherwise, why use assert() in the first place?) or remove the assert() because the assumption made is no longer valid (this has happened to me but not often and usually after code has changed, which is something you want, no?).

                                    3. 2

                                      You can do #define assert(p) if (!(p)) __builtin_unreachable() to keep the optimisation benefits! And MSVC has __assume which behaves similarly.

                                      1. 1

                                        Hmm… Interesting….

                                        Does it then elide the expression !(p)?

                                        Or does it impose the run time cost of evaluating !(p) and not the benefit of invoking abort()?

                                        1. 1

                                          Since __builtin_unreachable only exists to guide the compiler, p has to be an expression with no side effects, and then the compiler can optimise it out because you don’t use its result.

                                      2. 1

                                        I think this is an incorrectly framed question. C says that it’s not the compiler’s problem. You index past an array bound, perhaps you know what you are doing or perhaps not. The compiler is just supposed to do what you said. If you have indexed into another data structure by mistake or past the bound of allocated memory - that’s on the programmer ( BTW: I think opt-in bounds checked arrays would be great). It is unreasonable for the compiler to assume things that may be false. For example, if the programmer cautiously adds a check for overflow, I don’t want the compiler to assume that the index must be in bounds so the check can be discarded.

                                        1. 6

                                          C says that it’s not the compiler’s problem

                                          Actually the C standard says it’s not the compilers problem, it’s undefined behaviour and completely your problem.

                                          If you want it to have some weird arsed, but well defined behaviour, you need a different language standard.

                                          In C standardese, things that are “the compilers problem” are labelled clearly as “implementation defined”, things that are your problem are labelled “undefined behaviour”.

                                          perhaps you know what you are doing or perhaps not.

                                          Well, actually, you provably don’t know what you’re doing…. as the compiler and linker lays out the data structures in ram pretty much as they damn well feel like.

                                          Part of that for, example, like the struct padding and alignment is part of the ABI for that particular system, which is not part of the C standard, and most of that will change as you add or remove other data items and/or change their types. If you need to rely on such things, there are other (some non-standard) mechanisms, eg. unions types and packing pragmas.

                                          BTW: I think opt-in bounds checked arrays would be great

                                          gcc and clang now does have sanitizers to check that.

                                          However the C standard is sufficiently wishy-washy on a number of fronts, there are several corner cases that are uncheckable, and valgrind is then your best hope. Valgrind won’t help you, for example, if you index into another valid memory region or alignment padding.

                                          For example, if the programmer cautiously adds a check for overflow, I don’t want the compiler to assume that the index must be in bounds so the check can be discarded.

                                          How ever, if the compiler can prove that the check always succeeds, then the check is useless and the programmer has written useless code and the compiler rightly elides it.

                                          Modern versions of gcc will (if you have the warnings dialled up high enough, and annotated function attributes correctly) will warn you about tautologies and unreachable code.

                                          1. 1

                                            The C standard is not the C language. It is a committee report attempting to codify the language. It is not made up of laws of physics - it can be wrong and can change. My argument is that the standard is wrong. Feel free to disagree, but please don’t treat the current instance of the standard as if they were beyond discussion.

                                            In fact, I do want a different standard: one that is closer to my idea what the rules of the language should be in order to make the language useful, beautiful, and closer to the spirit of the design.

                                            The compiler and linker don’t have total freedom to change layouts even in the current standard - otherwise, for example, memcpy would not work. Note: “Except for bit-fields, objects are composed of contiguous sequences of one or more bytes, the number, order, and encoding of which are either explicitly specified or implementation-defined.”

                                            struct a{ int x[100];}; char *b = malloc(sizeof(int)*101; struct a *y = (struct a *)b; if(sizeof(struct a)) != sizeof(int)*100 ) panic(“this implementation of C won’t work for us\n”); …. do stuff … y->x[100] = checksum(y);

                                            But worse,

                                            message = readsocket(); for(i = 0; i < message->numberbytes; i++) if( i > MAX)use(m->payload[i]))

                                            if the compiler can assume the index is never greater than the array size and MAX is greater than array size, according to you it should be able to “optimize” away the check.

                                            How ever, if the compiler can prove that the check always succeeds, then the check is useless and the programmer has written useless code and the compiler rightly elides it.

                                            This is one of the key problems with UB. The compiler can assume there is no UB. Therefore the check is assumed unnecessary. Compilers don’t do this right now, but that’s the interpretation that is claimed to be correct. In fact, in many cases the compiler assumes that the code will not behave the way that the generated code does behave. This is nutty.

                                            1. 8

                                              The C standard is not the C language.

                                              Hmm. So what is The C Language?

                                              In the absence of the standard, there is no “C Language”, merely a collection of competing implementations of different languages, confusingly all named “C”.

                                              I don’t think calling a standard “Wrong” isn’t very helpful, as that would imply there exists some definition of Right.

                                              I rather call it “differing from all known implementations” or “unimplementable” or “undesirable” or “just plain bloody silly”.

                                              There is no One True Pure Abstract Universal C out there like an ancient Greek concept of Numbers.

                                              There are only the standard(s) and the implementations.

                                              In fact, I do want a different standard: one that is closer to my idea what the rules of the language should be in order to make the language useful, beautiful, and closer to the spirit of the design.

                                              Ah, the Joys of Standards! They are so useful, everybody wants their own one! ;-)

                                              Except for bit-fields, objects are composed of contiguous sequences of one or more bytes, the number, order, and encoding of which are either explicitly specified or implementation-defined.”

                                              Umm. So, yes, an array is a contiguous sequence, but we’re talking about indexing out of bounds of an array. So what is contiguous beyond that array?

                                              Answer 1 : Possibly empty padding to align the next object at the appropriate alignment boundary.

                                              Answer 2: Which is the next object? That is determined by the field order within a struct…. (with the alignment padding determined by the ABI), but if the array is not in a struct…. it’s all bets off as to which object the compiler linker chooses to place next.

                                              Hmm. Your example didn’t format nicely (nor was it valid syntax (missing parenthesis) so let me see if I can unravel that to see what you mean…

                                              struct a { 
                                                 int x[100];
                                              }; 
                                              char *b = malloc(sizeof(int)*101); 
                                              struct a *y = (struct a *)b; 
                                              
                                              if(sizeof(struct a)) != sizeof(int)*100 ) 
                                                  panic(“this implementation of C won’t work for us\n”); 
                                              
                                              …. do stuff … 
                                              
                                               y->x[100] = checksum(y);
                                              

                                              Hmm. Not sure what you’re trying to say, but try this one….

                                              #include <stdio.h>
                                              #include <stdint.h>
                                              
                                              struct {
                                                  char c[5];
                                                  uint32_t i;
                                              } s;
                                              
                                              uint64_t l;
                                              
                                              int main(void)
                                              {
                                                 printf( "sizeof(s)=%lu\n sizeof(c)=%lu\n sizeof(i)=%lu\n", sizeof(s),sizeof(s.c),sizeof(s.i));
                                                 printf( "address of s=%08lx\n address of l=%08lx\n diff = %ld\n", (uintptr_t)&s, (uintptr_t)&l, ((intptr_t)&s-(intptr_t)&l));
                                                 return 0;
                                              }
                                              

                                              Outputs…

                                              sizeof(s)=12                                                                                                                                                                                 
                                              sizeof(c)=5                                                                                                                                                                                 
                                              sizeof(i)=4                                                                                                                                                                                 
                                              address of s=00601050                                                                                                                                                                        
                                              address of l=00601048                                                                                                                                                                       
                                              diff = 8                                                                                                                                                                                    
                                              

                                              https://onlinegdb.com/B1Ut3031m

                                              1. 3

                                                In the absence of the standard, there is no “C Language”, merely a collection of competing implementations of different languages, confusingly all named “C”.

                                                What a weird idea. So much of the core code of the internet and infrastructure was written in an impossible language prior to the first ANSI standard. And it even worked!

                                                Ah, the Joys of Standards! They are so useful, everybody wants their own one! ;-)

                                                There is a standards process. It involves people presenting proposals for modifications and discussing their merits. There are changes ! That’s totally normal and expected.

                                                The moment C compilers began padding, every compiler added a “packed” attribute. The reason is that many C applications require that capability. Imagine ethernet packets with artisanal compiler innovative ordering. And those attributes are not in the standard - yet they exist all the same.

                                                Your example is not my example.

                                                1. 2

                                                  So much of the core code of the internet and infrastructure was written in an impossible language prior to the first ANSI standard. And it even worked!

                                                  Were actually written in whatever dialect was available on the day and worked for that machine, that compiler on that day.

                                                  And porting to a different machine, different compiler, different version of the compiler, was a huge pain in the ass.

                                                  I know.

                                                  I’ve done a hell of a lot of that over the decades.

                                                  Role on tighter standards please.

                                                  Yup, and most of them added a subtly different packed attribute, and since it was not terribly well documented and defined, I’ve had a fair amount of pain from libraries written (LWIP comes to mind), where at various points in their history got the packed attribute wrong, so it wasn’t portable from 32 to 64 bit.

                                                  Your example is not my example.

                                                  You example wasn’t formatted, and I didn’t quite understand the point of it. Can you format it properly and expand a bit on what you were sayign with that example?

                                      1. 9

                                        It’s a neat language that I hope to see more of. That said, I haven’t seen any evidence of this:

                                        “but the argument is that the increase in productivity and reduction of friction when memory-safe mechanisms are absent more than make up for the time lost in tracking down errors, especially when good programmers tend to produce relatively few errors.”

                                        I have seen studies show safe, high-level programming give productivity boosts. I’ve also seen a few that let programmers drop into unsafe, manual control where they want with that wrapped in a safe or high-level way. Combining these should probably be the default approach unless one can justify benefits of not doing so. Also, we’ve seen some nice case studies recently of game developers getting positive results trying Design-by-Contract and Rust. If author was right, such things would’ve slowed them down with no benefits. Similarly for the game developers that are using Nim with its controllable, low-latency GC.

                                        1. 4

                                          It’s not mentioned in the primer but the compiler has built in support for linting. You can access to the AST during compilation and stop the build, so rather than enforcing good practice by shoehorning it into types you can enforce good practice by directly checking for misuse.

                                          I do wonder if people will just end up badly implementing their own type systems on top of that though.

                                          1. 2

                                            That’s exactly the kind of thing I was describing in my response to akkartik. Neat stuff.

                                          2. 1

                                            You’re making some big claims here (even a normative statement), seemingly without much to back it up.

                                            I have seen studies show safe, high-level programming give productivity boosts.

                                            Too general. For some tasks high-level languages are clearly desirable, for others they clearly don’t work at all. The question at hand is which level of abstraction is desirable for writing/maintaining a large, complex codebase that has to reliable mangle a huge amount of data about 60 times per second. If a study does not replicate these conditions, it is worthless for answering the question.

                                            I’ve also seen a few that let programmers drop into unsafe, manual control where they want with that wrapped in a safe or high-level way. Combining these should probably be the default approach unless one can justify benefits of not doing so.

                                            Does writing your engine in C(++) and slapping Lua on top for the game logic count? Many games work like that, it pretty much is the default approach; no immature, unproven languages needed.

                                            Also, we’ve seen some nice case studies recently of game developers getting positive results trying Design-by-Contract and Rust.

                                            Unless a serious game in Rust has actually been released and enjoyed some success, we haven’t. A lone Rust enthusiast playing around for a few month and writing a blog post about it does not tell us a whole lot about how Rust might fare under realistic conditions.

                                            If author was right, such things would’ve slowed them down with no benefits.

                                            What makes you so sure they haven’t?

                                            Every time I read posts like yours I wonder if Rust evangelists and other C haters ever seriously ponder why people use and like C, why it became the giant on whose shoulders so much of modern software stands. I’m not claiming it all comes down to technical superiority, but I do think there is a reason C has stood the test of time like no other programming language in computing’s (admittedly not very long) history. And it certainly wasn’t for a lack of competition.

                                            Edit: I was reminded of Some Were Meant for C, which should be required reading for anyone developing a new “systems” language.

                                            1. 3

                                              “Every time I read posts like yours I wonder if Rust evangelists and other C haters ever seriously ponder why people use and like C, why it became the giant on whose shoulders so much of modern software stands.”

                                              It’s rare I get accused of not reading enough history or CompSci papers on programming. I post them here constantly. On the contrary, it seems most C programmers didn’t study the history or design of the language since most can’t even correctly answer “what language invented the philosophy of being relatively small, efficiency focused, and the ‘programmer is in control’ allowing direct manipulation of memory?” They’ll almost always say C sine their beliefs come from stories people told them instead of historical research. Correct answer was BCPL: the predecessor that C lifted its key traits off of. Both were designed almost exclusively due to extremely-limited hardware of the time. They included just what could run and compile on weak hardware modified with arbitrary, personal preferences of the creators. Here’s a summary of a presentation that goes paper by paper studying the evolution of those designs. The presentation is Vimeo link in references. Only thing I retract so far is “not designed for portability” since a C fan got me a reference where they claimed that goal after the fact.

                                              “You’re making some big claims here (even a normative statement), seemingly without much to back it up.”

                                              After accusing me of doing no research, you ironically don’t know about the prior studies comparing languages on traits like these. There were many studies done in research literature. Despite the variance in them, certain things were consistent. Of the high-level languages, they all killed C on productivity, defect rate, and maintenance. Ada usually beat C++, too, except in one study I saw. Courtesy of Derek Jones, my favorite C vs Ada study is this since it’s so apples to apples. Here’s one from patrickdlogan where they do same system in 10 languages with everything that’s been mainstream beating C in productivity. Smalltalk leaves all them in the dust. The LISP studies showed similar boost in development speed. Both supporting quick iterations, working with live image/data, and great debugging help with that. LISP also has its full-strength, easy-to-run macros with their benefits. Example of benefits of LISP-based OS and Smalltalk live-editing (search for “Apple” for part about Jobs’ visit).

                                              Hell, I don’t have to speculate to you given two people implemented C/C++ subset in a Scheme in relatively short time maintaining the benefits of both. Between that, Chez Scheme running on a Z80, and PreSceme alternative to C, that kind of shows that we’ve been able to do even what C does with more productivity, safer-by-default, easier debugging, and easier portability since somewhere from the 1980’s to mid-to-late-1990’s depending on what you desired. If a modern take, one could do something like ZL above in Racket Scheme or IDE-supported Common LISP. You’d get faster iterations and more productivity, esp extensibility, due to better design for these things. Good luck doing it with C’s macros and semantics. That combo is so difficult it just got a formal specification w/ undefined behavior a few years ago (KCC) despite folks wanting to do that for decades. The small LISP’s and Pascals had that stuff in the 1980-1990’s due to being designed for easier analysis. Other tech have been beating it on predictable concurrency such as Concurrent Pascal, Ada Ravenscar, Eiffel SCOOP, and recently Rust. Various DSL’s and parallel languages do easier parallel code. They could’ve been a DSL in the better-designed C which is approach some projects in Rust and Nim are taking due to macro support.

                                              So, we’re not knocking it over speculative weaknesses. It was a great bit of hackery back in the early 1970’s for getting work done on a machine that was a piece of shit using borrowed tech from a machine that was a bigger piece of shit. Hundreds of person-years of investment into compiler and processor optimizations have kept it pretty fast, too. None of this changes that it has bad design relative to competitors in many ways. None of it changes that C got smashed every time it went against another language balancing productivity, defect rate, and ease of non-breaking changes. The picture is worse when you realize that people building that ecosystem could’ve built a better language that FFI’d and compiled to C as an option to get its ecosystem benefits without its problems. That means what preserves C is basically a mix of inertia, social, and economic factors having about nothing to do with its “design.” And if I wanted performance, I’d look into synthesis, superoptimizers, and DSL’s designed to take advantage of modern, semi-parallel hardware. C fails on that, too, these days that most people code that stuff in assembly by hand. Those other things have recently been outperforming them, too, in some areas.

                                              There’s not a technical reason left for its superiority. It’s just social and economic factors at this point driving the use of A Bad Language for today’s needs. Better to build, use, and/or improve A Good Language using C only where you absolutely have to. Or want to if you simply like it for personal preference.

                                              @nullp0tr @friendlysock I responded here since it was original article that led to re-post of the other one. I’d just be reusing a lot of same points. I also countered the other article in the original submission akkartik linked to in that thread.

                                              1. 3

                                                It’s rare I get accused of not reading enough history or CompSci papers on programming.

                                                And I didn’t. I’m aware of your posts and interests. Rather, I’m accusing you of reading too much and doing too little. In the real world, C dominates. Yet studies have found only disadvantages. Clearly the studies are missing something fundamental. Instead of looking for that missing something, you keep pointing at the studies, as if they contain the answer. They cannot, because their conclusions fly in the face of reality.

                                                After accusing me of doing no research, you ironically don’t know about the prior studies comparing languages on traits like these.

                                                I’m aware of (some of) the research. Color me unimpressed. I don’t need a study to know that C is a crappy language for application development. Nor to figure out that C has serious warts and problems all over, even for OS development.

                                                What I would very much like to see more research on is why C keeps winning anyway. You chalk it up to “socio-economic factors” and call it a day. I call that a job not finished. It doesn’t explain why people use and like products built in C. It doesn’t explain why some very capable engineers defend C with a vigor that puts any Rust evangelist to shame[1]. It doesn’t explain what these socio-economic factors are, but they clearly matter a great deal. Otherwise, we agree, C wouldn’t be where it is today.[2]

                                                Legacy/being well-established is certainly one factor keeping C alive, but how did it become so dominant? As a computer history buff you might argue that it is because universities and their inhabitants had cheap access to UNIX systems back in the day and so students learned C and stuck to it when they graduated. There’s a lesson in that argument: having experience in a programming language is a very important factor in how productive you will be in it, which means throwing that experience away by using a very different language is a huge loss. Those who earn their bread and butter by having and using that experience will be very reluctant to do so. Most working programmers simply can/will not afford to learn a radically different language. It is an even bigger loss to society at large than to individuals because you loose teachers, senior engineers and others who pass on experience too. There are other network/feedback effects (ignoring the obvious technical ones), but I think you get the point.

                                                The consequence is that small incremental improvements on existing (and proven) technology are vastly preferable to top-down fundamental redesigns, even if that means the end result isn’t anywhere close to pretty. This has been proven again and again across industries and live in general. x86 is another prominent example.

                                                And then there is Go. Here’s a language that is built by people who understand C and the need to ease adoption. And look! People are actually using it! Go has easily replaced more C code on my machines than all the strongly typed, super-duper safe languages combined and probably in the internet-at-large too. And, predictably, the Haskell enthusiasts rage and shake their fists at it, because it isn’t the panacea they imagine their favorite strongly-typed language to be. Or in other words: because they do not understand, and are unwilling to learn, the most basic aspects about what keeps C alive.

                                                Anyway, I wanted to circle back to your original claim that memory-safety mechanisms do not inhibit productivity in game programming, but this is already pretty long and I’m hungry. Maybe I’ll write another comment to address that later.

                                                [1] And they make good on their claims by building things that are actually successful, instead of sticking to cheap talk about the theoretical advantages and showing off their skills at doing weird stuff to type systems.

                                                [2] That does not mean I don’t believe there are important technical aspects to C which are missing in languages intended to replace it. See the “Some were meant for C” paper.

                                                1. 0

                                                  “What I would very much like to see more research on is why C keeps winning anyway.”

                                                  It’s not “winning anyway” any more than Windows keeps winning on the desktop, .NET/Java in enterprise, PHP in web, and so on. It got the first mover advantage in its niche. It spread like wildfire via UNIX and other platforms that got huge. It was a mix of availability, hacker culture, and eventually open-source movement. Microsoft and IBM were using monopoly tactics suing or acquiring competitors that copied their stuff with different implementations or challenged them. Even companies like Borland with Pascals outperforming C on balance of productivity and speed saw the writing on the wall. Jumped on the bandwagon adding to it. The momentum and effects of several massive trends moving together who all found common ground in C at that time led to oligopolies of legacy code and decades of data locked into it. They have massive, locked-in base of code exposing C and C++. To work on it or add to it, the easiest route was learning C or C++ which increased the pool of developers even more. It’s self-reinforcing after a certain point.

                                                  I don’t have hard data on why non-UNIX groups made their choices. I know Apple had Pascal and LISP at one point. They eventually went with Objective-C for some reason with probably some C in there in the middle. IBM was using PL/S for some mainframe stuff and C for some other stuff. There’s gaps in what I can explain but each one I see is a company or group jumping on it. Then, it gets more talent, money, and compilers down the line. What explains it is Gabriel’s Worse is Better approach, herd mentality of crowds, network effects, and so on. You can’t replicate the success of C/C++ as is since it appeared at key moments in history that intersected. They won’t repeat. There will be similar opportunities where something outside a language is about to make waves where a new language can itself make waves by being central to or parasite off it. Economics, herd mentality, and network effects tell us it will work even better if it’s built on top of an existing ecosystem. A nice example is Clojure that built something pretty different on top of Java’s ecosystem. Plus all the stuff allowing seemless interoperability with C, building on Javascript in browsers, or for scripting integration with something like Apache/nginx.

                                                  So, it’s pretty clear from the lack of design to the enormous success/innovation to the stagnation of piles of code that C’s gone through some cycle of adoption driven by economics. There’s lasting lessons that successful alternatives are using. Once you know that, though, there’s not much more to learn out of necessity. Plenty for curiosity but the technical state of the art has far advanced. That’s about the best I can tell you about that part of your post after a long day. :)

                                                  “There’s a lesson in that argument: having experience in a programming language is a very important factor in how productive you will be in it, which means throwing that experience away by using a very different language is a huge loss.”

                                                  Now we’re in the “C is crap with certain benefits from its history that might still justify adoption” territory. This is essentially an economic argument saying you’ve invested time in something that will pay dividends or you don’t want to invest or have time to get a new language to that point. You see, if it’s designed right, this isn’t going to hurt you that much. The safe alternatives to C in language and libraries kept all that investment while making some stuff safe-by-default. There’s some languages that are just easy to understand but compile to C. Then, there’s those that are really different that still challenge your claim. The apples-to-apples study I linked on Ada and C specifically was concerned about their C developers doing worse in Ada due to a lack of experience. However, those developers did better since the language basically stopped a lot of problems that even experienced developers kept hitting in C since it couldn’t catch them. At least for that case, this concern was refuted even with a safe, systems language with a steep learning curve.

                                                  So, it’s not always true. It is worth considering carefully, though. Do note that my current advice for C alternatives is keeping close enough to C to capitalize on existing understanding, code, and compilers.

                                                  “Most working programmers simply can/will not afford to learn a radically different language.”

                                                  That’s a hypothesis that’s not proven. If anything, I think the evidence is strongly against you with programmers learning new languages and frameworks all the time to keep their skills relevant. Many are responding positively to the new groups of productive, safe languages such as Go, Rust, and Nim. Those fighting the borrow-checker in Rust about to quit usually chill when I tell them they can just use reference counting on the hard stuff till they figure it out. It’s a fast, safe alternative to Go at that point. If they want, they can also turn off the safety to basically make it a high-level, C alternative. There’s knobs to turn to reduce difficulty of using new, possibly-better tooling.

                                                  “People are actually using it! “

                                                  Go was a language designed by a multi-billion dollar corporation whose team had famous people. The corporation then pushed it strongly. Then, there was adoption. This model previously gave us humongous ecosystems for Java and then C#/.NET. They even gave Java a C-like syntax to increase adoption rate. Go also ironically was based on the simple, safe, GC’d approach of Niklaus Wirth that Pike experienced in Oberon-2. That was a language philosophy C users fought up to that point. Took a celebrity, a big company, and strong focus on tooling to get people to try what was essentially a C-like Oberon or ALGOL68. So, lasting lessons are getting famous people involved, have big companies with platform monopolies pushing it, make it look kind of like things they like, and strong investments in tooling as always.

                                                  “Anyway, I wanted to circle back to your original claim that memory-safety mechanisms do not inhibit productivity in game programming, but this is already pretty long and I’m hungry. Maybe I’ll write another comment to address that later.”

                                                  I’m interested in that. Remember, though, that I’m fine with being practical in an environment where high, resource efficiency takes priority over everything else. In that case, the approach would be safe-by-default with contracts for stuff like range checks or other preconditions. Tests generated from those contracts with automatic checks showing you the failures. If performance testing showed it too slow, then the checks in the fast path can be removed where necessary to speed it up. Throw more automated analysis, testing, or fuzzing at those parts to make up for it. If it’s a GC, there’s low-latency and real-time designs one might consider. My limited experience doing games a long time ago taught me memory pools help in some places. Hell, regions used in some safe languages sound a lot like them.

                                                  So, I’m advocating you use what safety you can by default dialing it down only as necessary to meet your critical requirements. The end result might be way better than, slightly better than, or equivalent to C. It might be worse if it’s some combo of a DSL and assembly for hardware acceleration which simply can have bugs just due to immaturity. Your very own example of Go plus recent ones with Rust show a lot of high-performance, latency-sensitive apps can go safe by default picking unsafety carefully.

                                          1. 2

                                            The netcode.io spec talks about crypto but it seems strange to me, using pubkey crypto for messages instead of doing a key exchange.

                                            Valve also recently published their netcode library. It’s not documented but does link to the QUIC crypto spec which might be worth reading.

                                            1. 43

                                              bing has no qualms hitting meta more than 5000 times in a 3 hour period

                                              We decided to take this drastic measure to protect Discourse sites out there from being attacked by Microsoft crawlers.

                                              One read every two seconds is not an attack, it’s totally irrelevant unless your software is many (at least 3?) orders of magnitude too slow.

                                              Maybe they should work on that before shifting the blame onto Microsoft.

                                              1. 10

                                                In C++ this is

                                                template< typename T >
                                                constexpr T max( T a, T b ) { return a > b ? a : b; }
                                                

                                                I know Linus is afraid of the typical/“modern” C++ crowd killing his codebase, but you don’t have to write C++ like that. You can write your normal C, just with templates/constexpr functions instead of junky macro hacks like this.

                                                1. 3

                                                  I was thinking the same thing.

                                                  There are plenty of arguments against C++, but it’s silly to rage against C++ and then allow code like this.

                                                1. 3

                                                  I suspect the hash function he uses is not ideal.

                                                  Multiplying integers is really surprisingly slow. On Skylake, you can start one r32 IMUL per cycle and you have to wait 4 cycles for the result. You can start four AND/OR/XORs per cycle with 1 cycle latency, so that’s 16 bitops in the same time as a single multiply.

                                                  1. 4

                                                    The multiplier is 31, which is a sign the hash function is designed to be implemented by bitshifting. That is, h * 31 is the same as (h << 5) - h. I use a similar DJB hash here:

                                                    https://github.com/google/rappor/blob/master/analysis/cpp/find_cliques.cc#L81

                                                    In that case the multiplier is 33, so you do (h << 5) + h.

                                                    I think Java has problems with shifts because it doesn’t have unsigned integers? But the JVM is pretty performance-minded so I imagine it should do “the right thing”.

                                                  1. 6

                                                    I ported my compressor to use AVX2. Most of it was find and replace because most SSE instructions have a same-but-twice-as-wide partner in AVX. Not all of it was that easy though, SSE has some instructions that work on the entire 128bit register (e.g. PSHUFB) while their AVX counterparts work on 2x128bits rather than 1x256bits. Took some staring to figure out which parts were actually broken and the fixes are ugly and slow but it works now.

                                                    The AVX implementation is somewhat faster than the SSE implementation, which I’m pretty sure makes it the fastest decoder in the world. Compression is not awesome but I have some experiments to try which may also end up making it faster as a bonus.

                                                    I’ve been having a lot of fun working on this and I’ve been thinking about how to move my career towards this kind of work too. I like algorithms and low level performance tweaking with more science than engineering. It seems to me that jobs like this simply do not exist (obviously they do but they’re extremely rare) so I’m investigating starting my own company to work on it. Going to try to schedule some phone calls with people that have taken similar paths and other people who might be interested in buying it and see what happens from there.

                                                    1. 4

                                                      Garbage in, garbage out. If you run the same test with

                                                      template< typename A, typename B >
                                                      struct Pair {
                                                          A a;
                                                          B b;
                                                      };
                                                      

                                                      in place of std::pair, you get

                                                      pair.cc: In function ‘int main()’:
                                                      pair.cc:14:28: error: use of deleted function ‘Pair<std::basic_string<char>, A>::Pair()’
                                                           Pair< std::string, A > p;
                                                                                  ^
                                                      pair.cc:8:8: note: ‘Pair<std::basic_string<char>, A>::Pair()’ is implicitly deleted because the default definition would be ill-formed:
                                                       struct Pair {
                                                              ^~~~
                                                      pair.cc:8:8: error: no matching function for call to ‘A::A()’
                                                      pair.cc:4:5: note: candidate: A::A(int)
                                                           A( int ) { }
                                                           ^
                                                      pair.cc:4:5: note:   candidate expects 1 argument, 0 provided
                                                      pair.cc:3:8: note: candidate: constexpr A::A(const A&)
                                                       struct A {
                                                              ^
                                                      pair.cc:3:8: note:   candidate expects 1 argument, 0 provided
                                                      pair.cc:3:8: note: candidate: constexpr A::A(A&&)
                                                      pair.cc:3:8: note:   candidate expects 1 argument, 0 provided
                                                      

                                                      This is a problem with the STL, not gcc.

                                                      1. 12

                                                        At home, I’ve been optimising my compression algorithm.

                                                        The encoder hasn’t received much attention yet, but I found a very nice trick for optimising one small part and it doubled the speed of the whole thing, putting it on par with LZ4. With tweaking it should beat LZ4 quite handily.

                                                        The decoder is where I’ve been focusing my efforts, and it destroys anything in the open source world. I’ve seen 60% faster with the same or a bit better compression than LZ4 on normal non-degenerate files, and I still have work to do. There’s definitely room for tweaking, and I want to try widening to AVX2. AVX is a bit derp though so I’m not sure if that will work out.

                                                        This is all particularly exciting because LZ4 has the world’s fastest practical LZ encoder (ruling out things like density) and I believe I’m in the same ballpark as the world’s fastest decoder (but with much less compression :<).

                                                        Next steps are to become #1 and figure out how to turn it into money.

                                                        1. 10

                                                          I believe there’s a ‘documentary’ called Silicon Valley that can be a good starting guide.

                                                          1. 2

                                                            I think they even had a real-world, case study toward the end involving life-like hands and cylindrical objects. Might be useful for bicycle games or something.

                                                          2. 2

                                                            Good luck with your project!

                                                            I’m observing compression-related subjects from the sidelines occasionally. (Lately I started providing some moderately recent Windows binaries for some compression-related tools, that are not widely available.)

                                                            Are you perhaps unaware of encode.ru forum? You can look there for some projects that were based on LZ4, like LZ4X or LZ5, and a lot of other interesting stuff, like promising RAZOR - strong LZ-based archiver for instance. You’ll find there knowledgeable people from the field and their insightful posts.

                                                            1. 1

                                                              Giving RAD a run for their money eh? :)

                                                              1. 1

                                                                Sort of. I’m trying to avoid taking them head on because I won’t win that. AFAIK they don’t put much effort into having super fast encoders, which leaves room for me to focus on end to end latency.

                                                            1. 2

                                                              This seems like spam to me?

                                                              They’ve copy pasted the advisory and put a very stupid title on it. “Ugly, perfect ten-rated bug”, “patch before they’re utterly p0wned”, “btw it’s a denial of service attack”.

                                                              No thanks.

                                                              1. 2

                                                                It’s The Register; the self-admitted British tabloid (like the Daily Mail) of the IT world. Sometimes they can produce a good article, othertimes it’s clickbait where you’re also expecting page 3 to be a naked woman.

                                                                1. 1

                                                                  This vulnerability can “allow the attacker to execute arbitrary code and obtain full control of the system,” it’s not just a DoS.

                                                                  https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180129-asa1

                                                                1. 4

                                                                  On my PC at least, the process name has parens around it, so you can parse it with the Lua pattern %b(). Fishing out the parent PID would be like %S+ %b() %S+ (%S+).

                                                                  I don’t think the problem here is that /proc/pid/stat is broken, rather the string parsing tools in libc are not good enough and shouldn’t be used.

                                                                  The OpenBSD guys already ripped the patterns code out of Lua for their web server (patterns.c/.h), so it’s simple to add it to your own code base.

                                                                  1. 9

                                                                    Assuming the rules for process names is the same as the rules for file names, a process name can contain unbalanced parens. What if a process is named for example hello) world? The string in /proc//stat would be (hello) world), and %b() would just find (hello). This doesn’t seem like something which could be fixed by smarter parsing, other than by reading from the end.

                                                                    1. 2

                                                                      Oh good point, %((.*)%) works, assuming they don’t add parens to any other fields.

                                                                    2. 2

                                                                      As noted in the original post, even using the parentheses does not guard against this. The only thing I can think of to safely use the current format is scanning the whole file into a buffer and searching for the last closing parenthesis, then taking everything from the first opening parenthesis to the last as executable name.

                                                                      This is also not specific to C or libc, this format is bad to parse (and the mistake easy to make) with any language.

                                                                    1. 3

                                                                      This looks racy to me, can someone explain where I’m going wrong?

                                                                      Thread A is the first to acquire the benephore, checks and increments the atomic variable, finds it is zero and proceeds without acquiring the semaphore.

                                                                      While Thread A is in the critical section Thread B arrives and acquires the benephore, finds the atomic variable to be non-zero so acquires the semaphore. Because it has the semaphore it proceeds into the critical section. Now there are two threads concurrently in the critical section.

                                                                      What prevents this scenario?

                                                                      1. 3

                                                                        I think you’re right, unless I’m missing something obvious.

                                                                        Worse still, if T1 enters the critical section and is followed by T2, if T1 now makes progress it will find benaphore_atom > 1 and call release_sem() on a semaphore it doesn’t hold. Which is probably either a crash or UB, who knows.

                                                                        I was missing something obvious.

                                                                        The semaphore, is initialized into an ‘unavailable’ state.

                                                                        When Thread B attempts to acquire the newly initialized semaphore, it blocks as the semaphore is in its ‘unavailable’ state. Thread A later finishes up in its Critical Section, and seeing that benaphore_atom > 1 it increments the semaphore, allowing Thread B to make progress.

                                                                        At the end of this execution, T2 sees !(benaphore_atom > 1) and continues without marking the semaphore as available.

                                                                        1. 1

                                                                          Semaphores don’t provide mutual exclusion.

                                                                          You use them to e.g. count the number of items in a queue and wake up enough threads to process them. Then those threads use a separate mutex to synchronise any operations on the queue.

                                                                        1. 5

                                                                          If you’re benchmarking a really small piece of code in a loop, you also need to be aware that

                                                                          1. you can spend 50% of the time per iteration on loop overhead if you don’t unroll and
                                                                          2. CPUs execute multiple instructions in parallel, so if you aren’t careful measuring the time to do 1000 iterations and dividing by 1000 will underestimate the time to run one lone iteration (throughput vs latency)
                                                                          1. 6

                                                                            I’m starting work on proper FPS networking for my game. A lot of the concepts have clicked in my head lately so I feel ready to work on it, but it’s so crazy hard. There’s tons of tricky stuff I need to implement and when I sit down to start on it I just have no idea what to type. It feels like I’m programming for the first time again.

                                                                            So for the moment I’m doing mostly random exploration and hoping I find something reasonable. Wish me luck!

                                                                            1. 30

                                                                              All of them:

                                                                              The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                                              All of them:

                                                                              The fact that they waste so much effort on incremental builds when the compilers should really be so fast that you don’t need them. You should never have to make clean because it miscompiled, and the easiest way to achieve that is to build everything every time. But our compilers are way too slow for that.

                                                                              Virtually all of them:

                                                                              The build systems that do incremental builds almost universally get them wrong.

                                                                              If I start on branch A, check out branch B, then switch back to branch A, none of my files have changed, so none of them should be rebuilt. Most build systems look at file modified times and rebuild half the codebase at this point.

                                                                              Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                                                                              Virtually all of them:

                                                                              Making me write a build spec in something that isn’t a normal good programming language. The build logic for my game looks like this:

                                                                              if we're on Windows, build the server and all the libraries it needs
                                                                              if we're on OpenBSD, don't build anything else
                                                                              build the game and all the libraries it needs
                                                                              if this is a release build, exit
                                                                              build experimental binaries and the asset compiler
                                                                              if this PC has the release signing key, build the sign tool
                                                                              

                                                                              with debug/asan/optdebug/release builds all going in separate folders. Most build systems need insane contortions to express something like that, if they can do it at all,

                                                                              My build system is a Lua script that outputs a Makefile (and could easily output a ninja/vcxproj/etc). The control flow looks exactly like what I just described.

                                                                              1. 15

                                                                                The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                                                I disagree. Making the build system part of the language takes away too much flexibility. Consider the build systems in XCode, plain Makefiles, CMake, MSVC++, etc. Which one is the correct one to standardize on? None of them because they’re all targeting different use cases.

                                                                                Keeping the build system separate also decouples it from the language, and allows projects using multiple languages to be built with a single build system. It also allows the build system to be swapped out for a better one.

                                                                                Codebases easily fit in RAM …

                                                                                Yours might, but many don’t and even if most do now, there’s a very good chance they didn’t when the projects started years and years ago.

                                                                                Making me write a build spec in something that isn’t a normal good programming language.

                                                                                It depends on what you mean by “normal good programming language”. Scons uses Python, and there’s nothing stopping you from using it. I personally don’t mind the syntax of Makefiles, but it really boils down to personal preference.

                                                                                1. 2

                                                                                  Minor comment is that the codebase doesn’t need to fit into ram for you to hash it. You only need to store the current state of the hash function and can handle files X bytes at a time.

                                                                                2. 14

                                                                                  When I looked at this thread, I promised myself “don’t talk about Nix” but here I am, talking about Nix.

                                                                                  Nix puts no effort in to incremental builds. In fact, it doesn’t support them at all! Nix uses the hashing mechanism you described and a not terrible language to describe build steps.

                                                                                  1. 11

                                                                                    The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                                                    I’m not sure if I would agree with this. Wouldn’t it just make compilers more complex, bigger and error prone (“anti-unix”, if one may)? I mean, in some cases I do appriciate it, like with go’s model of go build, go get, go fmt, … but I wouldn’t mind if I had to use a build system either. My main issue is the apparent nonstandard-ness between for example go’s build system and rust’s via cargo (it might be similar, I haven’t really ever used rust). I would want to be able to expect similar, if not the same structure, for the same commands, but this isn’t necessarily given if every compiler reimplements the same stuff all over again.

                                                                                    Who knows, maybe you’re right and the actual goal should be create a common compiler system, that interfaces to particular language definitions (isn’t LLVM something like this?), so that one can type compile prog.go, compile prog.c and compile prog.rs and know to expect the same structure. Would certainly make it easier to create new languages…

                                                                                    1. 2

                                                                                      I can’t say what the parent meant, but my thought is that a blessed way to lay things out and build should ship with the primary tooling for the language, but should be implemented and designed with extensibility/reusability in mind, so that you can build new tools on top of it.

                                                                                      The idea that compilation shouldn’t be a special snowflake process for each language is also good. It’s a big problem space, and there may well not be one solution that works for every language (compare javascript to just about anything else out there), but the amount of duplication is staggering.

                                                                                      1. 1

                                                                                        Considering how big compilers/stdlibs are already, adding a build system on top would not make that much of a difference.

                                                                                        The big win is that you can download any piece of software and build it, or download a library and just add it to your codebase. Compare with C/C++ where adding a library is often more difficult than writing the code yourself, because you have to figure out their (often insane) build system and integrate it with your own, or figure it out then ditch it and replace it with yours

                                                                                      2. 8

                                                                                        +1 to all of these, but especially the point about the annoyance of having to learn and use another, usually ad-hoc programming language, to define the build system. That’s the thing I dislike the most about things like CMake: anything even mildly complex ends up becoming a disaster of having to deal with the messy, poorly-documented CMake language.

                                                                                        1. 3

                                                                                          Incremental build support goes hand in hand with things like caching type information, extremely useful for IDE support.

                                                                                          I still think we can get way better at speeding up compilation times (even if there’s always the edge cases), but incremental builds are a decent target to making compilation a bit more durable in my opinion.

                                                                                          Function hashing is also just part of the story, since you have things like inlining in C and languages like Python allow for order-dependent behavior that goes beyond code equality. Though I really think we can do way better on this point.

                                                                                          A bit ironically, a sort of unified incremental build protocol would let compilers avoid incremental builds and allow for build systems to handle it instead.

                                                                                          1. 2

                                                                                            I have been compiling Chromium a lot lately. That’s 77000 mostly C++ (and a few C) files. I can’t imagine going through all those files and hashing them would be fast. Recompiling everything any time anything changes would probably also be way too slow, even if Clang was fast and didn’t compile three files per second average.

                                                                                            1. 4

                                                                                              Hashing file contents should be disk-io-bound; a couple of seconds, at most.

                                                                                              1. 3

                                                                                                You could always do a hybrid approach: do the hash check only for files that have a more-recent modified timestamp.

                                                                                              2. 1

                                                                                                Do you use xmake or something else? It definitely has a lot of these if cascades.

                                                                                                1. 1

                                                                                                  It’s a plain Lua script that does host detection and converts lines like bin( "asdf", { "obj1", "obj2", ... }, { "lib1", "lib2", ... } ) into make rules.

                                                                                                2. 1

                                                                                                  Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                                                                                                  Unless your build system is a daemon, it’d have to traverse the entire tree and hash every relevant file on every build. Coming back to a non-trivial codebase after the kernel stopped caching files in your codebase will waste a lot of file reads, which are typically slow on an HDD. Assuming everything is on an SSD is questionable.