1. 3

    This is silly drama about something we already have far too many submissions for.

    1. 23

      This is a bit disappointing. It feels a bit like we are walking into the situation OpenGL was built to avoid.

      1. 7

        To be honest we are already in that situation.

        You can’t really use GL on mac, it’s been stuck at D3D10 feature level for years and runs 2-3x slower than the same code under Linux on the same hardware.

        It always seemed like a weird decision from Apple to have terrible GL support, like if I was going to write a second render backend I’d probably pick DX over Metal.

        1. 6

          I remain convinced that nobody really uses a Mac on macOS for anything serious.

          And why pick DX over Metal when you can pick Vulkan over Metal?

          1. 3

            Virtually no gaming or VR is done on a mac. I assume the only devs to use Metal would be making video editors.

            1. 1

              This is a bit pedantic, but I play a lot of games on mac (mainly indie stuff built in Unity, since the “porting” is relatively easy), and several coworkers are also mac-only (or mac + console).

              Granted, none of us are very interested in the AAA stuff, except a couple of games. But there’s definitely a (granted, small) market for this stuff. Luckily stuff like Unity means that even if the game only sells like 1k copies it’ll still be a good amount of money for “provide one extra binary from the engine exporter.”

              The biggest issue is that Mac hardware isn’t shipping with anything powerful enough to run most games properly, even when you’re willing to spend a huge amount of money. So games like Hitman got ported but you can only run it on the most expensive MBPs or iMac Pros. Meanwhile you have sub-$1k windows laptops which can run the game (albeit not super well)

            2. 2

              I think Vulkan might have not been ready when Metal was first skecthed out – and Apple does not usually like to compromise on technology ;)

              1. 2

                My recollection is that Metal appeared first (about June 2014), Mantle shipped shortly after (by a coupe months?), DX12 shows up mid-2015 and then Vulkan shows up in February 2016.

                I get a vague impression that Mantle never made tremendous headway (because who wants to rewrite their renderer for a super fast graphics API that only works on the less popular GPU?) and DX12 seems to have made surprisingly little (because targeting an API that doesn’t work on Win7 probably doesn’t seem like a great investment right now, I guess? Current Steam survey shows Win10 at ~56% and Win7+8 at about 40% market share among people playing videogames.)

                1. 2

                  Mantle got heavily retooled into Vulkan, IIRC.

                  1. 1

                    And there was much rejoicing. ♥

        1. 4

          This whole area has been exercising my brain recently.

          As much as I hate the C standard committee’s lack of courage in defining behaviour, as often a simple decision, even if controversial will resolve it.

          However, here is one that is sort of unresolvable.

          What behaviour should a program have that indexes beyond the bounds of an array?

          There is no way a standard can prescribe what the result will be.

          It must be undefined.

          So the obvious thing to do, as Pascal did, is do bounds checking and have a defined behaviour.

          That imposes substantial runtime costs in CPU and memory, so users do switch it off…..

          So what should the behaviour be?

          One reasonable assumption a compiler writer can make is that there is no way the programmer can intend to index out of bounds, so I can assume that the index is less than the bound and generate machine code accordingly.

          You might say, these newfangled optimizations are the problem… no they aren’t.

          Compilers have been relaying out data in memory according what they think best for decades.

          Where this whole thing is driving me nuts is around asserts. See this thread I started here… https://readlist.com/lists/gcc.gnu.org/gcc-help/7/39051.html

          Asserts, if they are compiled in, tell the compiler (without getting involved in UB optimizations) that if expression is false, then everything down stream is not reachable…. so it analyzes under the assumption the expression is true.

          However it doesn’t attempt to warn you if it finds a code path where the expression is false, and completely redoes it’s optimization without that assumption if you compile the assert out.

          1. 4

            What behaviour should a program have that indexes beyond the bounds of an array?

            There is no way a standard can prescribe what the result will be.

            It must be undefined.

            This gets to the heart of the matter, I think. Part of the issue is people confuse “the language standard” with “what compilers do”. The language says it is undefined behaviour for an out-of-bounds array access to occur, or for signed integers to have their value range exceeded, but there’s no reason why compilers can’t generate code which will explicitly detect these situations and throw out an error message (and terminate).

            So why don’t compilers generally do that by default? Because C is used in performance critical code where these checks have a cost which is considered significant. And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

            If you do value deterministic behaviour on program error and are willing to sacrifice some performance to get it, the obvious solution is to use a language which provides that, i.e. don’t use C. But that’s not a justification to criticise the whole concept of undefined behaviour in C.

            1. 4

              There is a false choice between inefficient code with run time bounds checking and compiler “optimizations” that break working code. I love the example in http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_2015_submission_29.pdf where the GCC developers introduce a really stupid UB based “optimization” that broke working code and then found, to their horror, that it broke a benchmark. So they disabled it for the benchmark.

              And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

              Great. Let’s see an example.

              But that’s not a justification to criticise the whole concept of undefined behaviour in C.

              I think this attitude comes from a fundamental antipathy to the design of C or a basic misunderstanding of how it is used. C is not Java or Swift - and not because its designers were stupid or mired in archaic technology.

              1. 4

                There is a false choice between inefficient code with run time bounds checking and compiler “optimizations” that break working code

                Optimisations don’t break working code. They cause broken code to have different observable behaviour.

                And, despite the claims in this article, there are cases where trivial optimisations such as assuming that signed integer arithmetic operations won’t overflow can lead to significant speedups (it’s just that these cases are not something as trivial as a single isolated loop).

                Great. Let’s see an example.

                I don’t have a code example to hand, and as I said they’re not trivial, but that doesn’t mean it’s not true. Since it can eliminate whole code paths, it can affect the efficacy of for example value range propagation, affect inlining decisions, and have other flow-on effects.

                I think this attitude comes from a fundamental antipathy to the design of C or a basic misunderstanding of how it is used

                I disagree.

                1. -1

                  Optimisations don’t break working code. They cause broken code to have different observable behaviour.

                  That’s a legalistic answer. The code worked as expected and produced the correct result. The “optimization” caused it to misfunction.

                  I don’t have a code example to hand

                  Apparently nobody does. So the claimed benefit is just hand waving.

                  I disagree.

                  The thinking of Lattner is indicative. He agrees that compiler behavior using the UB loophole makes C a minefield. His solution is to advocate Swift. People who are hostile to the use of C should not be making these decisions.

                  1. 5

                    That’s a legalistic answer.

                    Alas, in absence of “legalistic answers”, the only definition of C is either…

                    • An implementation of C is a valid implementation iff every program in a Blessed Set of programs compile and runs successfully and outputs exactly the same values.

                      or

                    • An implementation of C is a valid implementation iff, every program that compiles and runs successfully on The One True Blessed C Compiler, compiles and runs and outputs exactly the same values AND every program that fails to compile on The One True Blessed Compiler, fails to compile on the candidate compiler.

                    What sort of C are you envisioning?

                    Those may be appropriate ways to progress, but that is a different language and probably should be called something other than C.

                    1. 3

                      Apparently nobody does. So the claimed benefit is just hand waving

                      Again, I don’t agree.

                      1. 1

                        You can disagree all you want, but you also seem to be unable to produce any evidence.

                        1. 3

                          You can disagree all you want, but you also seem to be unable to produce any evidence.

                          I have high confidence that I could produce, given some time, an example of code which compiled to say 20 instructions if integer overflow were defined and just 1 or 2 otherwise, and probably more by abusing the same technique repeatedly, but you might then claim it wasn’t representative of “real code”. And then if I really wanted to satisfy you I would have to find some way to trawl through repositories to identify some piece of code that exhibited similar properties. It’s more work than I care to undertake to prove my point here, and so I suppose you have a right to remain skeptical.

                          On the other hand, I have at least explained (even if only very briefly) how small optimisations such as assuming that integer arithmetic operations won’t overflow could lead to significant differences in code generation, beyond simple exchanging of instructions. You’ve given no argument as to why this couldn’t be the case. So, I don’t think there’s any clearly stronger argument on either side.

                          1. 0

                            I have high confidence that I could produce, given some time, an example of code which compiled to say 20 instructions if integer overflow were defined and just 1 or 2 otherwise

                            I have no confidence of this, and it would be a completely uninteresting optimization in any case.

                            On the other hand, I have at least explained (even if only very briefly) how small optimisations such as assuming that integer arithmetic operations won’t overflow could lead to significant differences in code generation, beyond simple exchanging of instructions.

                            Not really. You are omitting a single instruction that almost certainly costs no cycles at all in a modern pipelined processor. Balance that against putting minefields into the code - and note there is no way in C to check for this condition. The tradeoff is super unappealing.

                            1. 2

                              Not really. You are omitting a single instruction

                              No, I was not talking about omitting a single instruction.

              2. 2

                With assert(), you are telling the compiler that at this point, this is true. The compiler is trusting your assertion of the truth at that point.

                Also, if you compile with -DNDEBUG -O3 you will get the warning:

                [spc]saltmine:/tmp>gcc -std=c99 -Wall -Wextra -pedantic -DNDEBUG -O3 c.c
                c.c: In function ‘main’:
                c.c:7:20: warning: ‘*((void *)&a+10)’ is used uninitialized in this function [-Wuninitialized]
                c.c:13:8: note: ‘a’ was declared here
                [spc]saltmine:/tmp>gcc -std=c99 -Wall -Wextra -pedantic -O3 c.c
                [spc]saltmine:/tmp>
                
                1. 2

                  No, that is a meaningless statement.

                  The compiler doesn’t even see an assert statement, let alone “trust it”.

                  It is a macro that gets expanded to “plain old code” at preprocessor time, so depending on NDEBUG settings it expands either to something like if(!(exp))abort() or nothing.

                  What the compiler does trust is the __attribute_((__noreturn__)) on the abort() function.

                  1. 1

                    My file:

                    #include <assert.h>
                    
                    int foo(int x)
                    {
                      assert(x >= 0);
                      return x + 5;
                    }
                    

                    My file after running it through the C preprocessor:

                    # 1 "x.c"
                    # 1 "/usr/include/assert.h"
                     
                     
                    
                     
                     
                     
                    
                     
                    # 12
                    
                    # 15
                    
                    #ident      "@(#)assert.h   1.10    04/05/18 SMI"
                    
                    # 21
                    
                    # 26
                    extern void __assert(const char *, const char *, int);
                    # 31
                    
                    # 35
                    
                    # 37
                    
                     
                    # 44
                    
                    # 46
                    
                    # 52
                    
                    # 63
                    
                    # 2 "x.c"
                    
                    int foo(int x)
                    {
                       ( void ) ( ( x >= 0 ) || ( __assert ( "x >= 0" , "x.c" , 5 ) , 0 ) );
                      return x + 5;
                    }
                    #ident "acomp: Sun C 5.12 SunOS_sparc 2011/11/16"
                    

                    Not an __atttribute__ to be found. This C compiler can now generate code as if x is never a negative value.

                    1. 1

                      #include <assert.h> int foo(int x) { assert(x >= 0); return x + 5; }

                      Can you copy paste the assembly output? (I read sparc asm as part of my day job….)

                      I’d be interested to see is it is treating __assert() as anything other than a common or garden function.

                      1. 1

                        I’m not sure what it’ll prove, but okay:

                            cmp     %i0,0
                            bge     .L18
                            nop
                        
                        .L19:
                            sethi   %hi(.L20),%o0
                            or      %o0,%lo(.L20),%o0
                            add     %o0,8,%o1
                            call    __assert
                            mov     6,%o2
                            ba      .L17
                            nop
                        
                            ! block 3
                        .L18:
                            ba      .L22
                            mov     1,%i5
                        
                            ! block 4
                        .L17:
                            mov     %g0,%i5
                        
                            ! block 5
                        .L22:
                        
                        !    7    return x + 5;
                        
                            add     %i0,5,%l0
                            st      %l0,[%fp-4]
                            mov     %l0,%i0
                            jmp     %i7+8
                            restore
                        
                            ! block 6
                        .L12:
                            mov     %l0,%i0
                            jmp     %i7+8
                            restore
                        

                        I did not specify any optimizations, and from what I can tell, it calls a function called __assert().

                        1. 1

                          TL;DR; The optimiser for this compiler is crap. And it isn’t treating __assert() as special / noreturn.

                          int foo(int x)
                          {
                             ( void ) ( ( x >= 0 ) || ( __assert ( "x >= 0" , "x.c" , 5 ) , 0 ) );
                            return x + 5;
                          }
                          
                              ;; x is register %i0
                              cmp     %i0,0               ; Compare x with 0
                              bge     .L18                ; If it is large branch to .L18
                              nop                         ; Delay slot. Sigh sparc pipelining is makes debugging hard.
                          
                          ;;; This is the "call assert" branch. gcc has function __attribute__((cold)) or
                          ;;; __builtin_expect() to mark this as the unlikely path.
                          .L19:
                              sethi   %hi(.L20),%o0       
                              or      %o0,%lo(.L20),%o0
                              add     %o0,8,%o1
                              call    __assert
                              mov     6,%o2               ;Delay slot again
                              ba      .L17                ; Branch absolute to .L17
                              nop                         ;Delay slot
                          
                              ;; Really? Is this optimized at all?
                              ! block 3
                          .L18:
                              ba      .L22                ; Branch absolute to .L22!
                              mov     1,%i5               ; put 1 in %i5
                          
                          ;;; Seriously? Is this thing trying to do it the hard way?
                          ;;; The assert branch sets %i5 to zero.
                              ! block 4
                              .L17:
                              ;; Fun fact. %g0 is the sparc "bit bucket" reads as zero, ignores anything written to it.
                              mov     %g0,%i5             
                          
                              ! block 5
                              ;; Falls through. ie. Expected to come 
                              ;; out of __assert() *hasn't treated __assert as noreturn!*
                          
                              ;; Joins with the x>=0 branch
                          .L22:
                          
                          !    7    return x + 5;
                              ;; Local register %l0 is x + 5
                              add     %i0,5,%l0
                              st      %l0,[%fp-4]         ;WTF? Has this been inlined into a larger block of code?
                              mov     %l0,%i0             ;WTF? as above?
                              jmp     %i7+8               ;Return to calling addres.
                              restore                     ;Unwind sparc register windowing.
                          
                              ;; WTF? No reference to label .L12
                              ! block 6
                          .L12:
                              mov     %l0,%i0
                              jmp     %i7+8
                              restore
                          
                  2. 1

                    Actually, that isn’t quite what happens….

                    Actually, it’s “Just Another Macro” which, very approximately, expands to …

                     if( !(exp)) abort();
                    

                    …where abort() is marked __attribute__((noreturn));

                    Which is almost, but not quite what one would want….

                    As the compiler uses the noreturn attribute to infer that if !exp, then rest of code is unreachable, therefore for rest of code exp is true.

                    Alas, I have found that it doesn’t, if it finds a path for which exp is false, warn you that you will abort!

                    I certainly feel there is room for compiler and optimizer writers work with design by contract style programmers to have a “mutually beneficial” two way conversation with the programmers when they write asserts.

                    1. 0

                      Which is almost, but not quite what one would want….

                      I’m not sure I understand you. assert() will abort if the expression given is false. That’s what it does. It also prints where the expression was (it’s part of the standard). If you don’t want to abort, don’t call assert(). If you expect that assert() is a compile-time check, well, it’s not.

                      I certainly feel there is room for compiler and optimizer writers work with design by contract style programmers to have a “mutually beneficial” two way conversation with the programmers when they write asserts.

                      There’s only so far that can go though. Put your foo() function in another file, and no C compiler can warn you.

                      assert() is also a standard C function, which means the compiler can have built-in knowledge of its semantics (much like a C compiler can replace a call to memmove() with inline assembly). The fact that GCC uses its __attribute__ extension for this doesn’t apply to all other compilers.

                      1. 2

                        That’s the other bit of Joy about C.

                        There are two entirely separate things….

                        The compiler.

                        And the standard library.

                        gcc works quite happily with several entirely different libc’s.

                        assert.h is part of libc, not the compiler.

                        How assert() is implemented is the domain of the libc implementer not the compiler writer.

                        I have poked at quite a few different compilers and even more libc’s…. as I have summarised is how all I have looked at are doing things. (Although some don’t have a concept of “noreturn” so can’t optimize based on that)

                        Which compiler / libc are you looking at?

                        1. 3

                          The compiler that comes with Solaris right now.

                          You can’t have a standard C compiler without the standard C library. I can get a compiler that understands, say, C99 syntax, but unless it comes with the standard C library, it can’t be called a compliant C99 compiler. The standard covers both the language and the library. I’m reading the C99 standard right now, and here’s an interesting bit:

                          Each library function is declared, with a type that includes a prototype, in a header, (182) whose contents are made available by the #include preprocessing directive.

                          And footnote 182 states:

                          (182) A header is not necessarily a source file, nor are the < and > delimited sequences in header names necessarily valid source file names.

                          To me, that says the compiler can have knowledge of the standard functions. Furthermore:

                          Any function declared in a header may be additionally implemented as a function-like macro defined in the header … Likewise, those function-like macros described in the following subclauses may be invoked in an expression anywhere a function with a compatible return type could be called.(187)

                          (187) Because external identifiers and some macro names beginning with an underscore are reserved, implementations may provide special semantics for such names. For example, the identifier _BUILTIN_abs could be used to indicate generation of in-line code for the abs function. Thus, the appropriate header could specify #define abs(x) _BUILTIN_abs(x) for a compiler whose code generator will accept it. In this manner, a user desiring to guarantee that a given library function such as abs will be a genuine function may write #undef abs whether the implementation’s header provides a macro implementation of abs or a built-in implementation. The prototype for the function, which precedes and is hidden by any macro definition, is thereby revealed also.

                          So the compiler can absolutely understand the semantics of standard C calls and treat them specially. Whether a C compiler does so is implementation defined. And good luck writing offsetof() of setjmp()/longjmp() portably (spoiler: you can’t—they’re tied to both the compiler and architecture).

                          So, getting back to assert() and your issues with it. Like I said, the compilers knows (whether it’s via GCC’s __attribute__(__noreturn__) or because the compiler has built-in knowledge of the semantics of assert()) that the expression used must be true and can thus optimize based on that information, much like it can remove the if statement and related code:

                          const int debug = 0;
                          
                          {
                            int x = debug;
                          
                            if (x)
                            {
                              fprintf(stderr,"here we are!\n");
                              exit(33);
                            }
                            // ...
                          }
                          

                          even through x, because debug is constant, x is loaded with a constant, and not modified prior to the if statement. Your wanting a warning about an invalid index to an array whose index is used in assert() is laudable, but to the compiler, you are telling it “yes, this is fine. No complaints please.” Compile the same code with NDEBUG defined, the assert() goes away (from the point of view of the compiler phase) and the diagnostic can be issued.

                          Yes, it sucks. But that’s the rational.

                          The intent is you run the code, you get the assert, you fix the code (otherwise, why use assert() in the first place?) or remove the assert() because the assumption made is no longer valid (this has happened to me but not often and usually after code has changed, which is something you want, no?).

                  3. 2

                    You can do #define assert(p) if (!(p)) __builtin_unreachable() to keep the optimisation benefits! And MSVC has __assume which behaves similarly.

                    1. 1

                      Hmm… Interesting….

                      Does it then elide the expression !(p)?

                      Or does it impose the run time cost of evaluating !(p) and not the benefit of invoking abort()?

                      1. 1

                        Since __builtin_unreachable only exists to guide the compiler, p has to be an expression with no side effects, and then the compiler can optimise it out because you don’t use its result.

                    2. 1

                      I think this is an incorrectly framed question. C says that it’s not the compiler’s problem. You index past an array bound, perhaps you know what you are doing or perhaps not. The compiler is just supposed to do what you said. If you have indexed into another data structure by mistake or past the bound of allocated memory - that’s on the programmer ( BTW: I think opt-in bounds checked arrays would be great). It is unreasonable for the compiler to assume things that may be false. For example, if the programmer cautiously adds a check for overflow, I don’t want the compiler to assume that the index must be in bounds so the check can be discarded.

                      1. 6

                        C says that it’s not the compiler’s problem

                        Actually the C standard says it’s not the compilers problem, it’s undefined behaviour and completely your problem.

                        If you want it to have some weird arsed, but well defined behaviour, you need a different language standard.

                        In C standardese, things that are “the compilers problem” are labelled clearly as “implementation defined”, things that are your problem are labelled “undefined behaviour”.

                        perhaps you know what you are doing or perhaps not.

                        Well, actually, you provably don’t know what you’re doing…. as the compiler and linker lays out the data structures in ram pretty much as they damn well feel like.

                        Part of that for, example, like the struct padding and alignment is part of the ABI for that particular system, which is not part of the C standard, and most of that will change as you add or remove other data items and/or change their types. If you need to rely on such things, there are other (some non-standard) mechanisms, eg. unions types and packing pragmas.

                        BTW: I think opt-in bounds checked arrays would be great

                        gcc and clang now does have sanitizers to check that.

                        However the C standard is sufficiently wishy-washy on a number of fronts, there are several corner cases that are uncheckable, and valgrind is then your best hope. Valgrind won’t help you, for example, if you index into another valid memory region or alignment padding.

                        For example, if the programmer cautiously adds a check for overflow, I don’t want the compiler to assume that the index must be in bounds so the check can be discarded.

                        How ever, if the compiler can prove that the check always succeeds, then the check is useless and the programmer has written useless code and the compiler rightly elides it.

                        Modern versions of gcc will (if you have the warnings dialled up high enough, and annotated function attributes correctly) will warn you about tautologies and unreachable code.

                        1. 1

                          The C standard is not the C language. It is a committee report attempting to codify the language. It is not made up of laws of physics - it can be wrong and can change. My argument is that the standard is wrong. Feel free to disagree, but please don’t treat the current instance of the standard as if they were beyond discussion.

                          In fact, I do want a different standard: one that is closer to my idea what the rules of the language should be in order to make the language useful, beautiful, and closer to the spirit of the design.

                          The compiler and linker don’t have total freedom to change layouts even in the current standard - otherwise, for example, memcpy would not work. Note: “Except for bit-fields, objects are composed of contiguous sequences of one or more bytes, the number, order, and encoding of which are either explicitly specified or implementation-defined.”

                          struct a{ int x[100];}; char *b = malloc(sizeof(int)*101; struct a *y = (struct a *)b; if(sizeof(struct a)) != sizeof(int)*100 ) panic(“this implementation of C won’t work for us\n”); …. do stuff … y->x[100] = checksum(y);

                          But worse,

                          message = readsocket(); for(i = 0; i < message->numberbytes; i++) if( i > MAX)use(m->payload[i]))

                          if the compiler can assume the index is never greater than the array size and MAX is greater than array size, according to you it should be able to “optimize” away the check.

                          How ever, if the compiler can prove that the check always succeeds, then the check is useless and the programmer has written useless code and the compiler rightly elides it.

                          This is one of the key problems with UB. The compiler can assume there is no UB. Therefore the check is assumed unnecessary. Compilers don’t do this right now, but that’s the interpretation that is claimed to be correct. In fact, in many cases the compiler assumes that the code will not behave the way that the generated code does behave. This is nutty.

                          1. 8

                            The C standard is not the C language.

                            Hmm. So what is The C Language?

                            In the absence of the standard, there is no “C Language”, merely a collection of competing implementations of different languages, confusingly all named “C”.

                            I don’t think calling a standard “Wrong” isn’t very helpful, as that would imply there exists some definition of Right.

                            I rather call it “differing from all known implementations” or “unimplementable” or “undesirable” or “just plain bloody silly”.

                            There is no One True Pure Abstract Universal C out there like an ancient Greek concept of Numbers.

                            There are only the standard(s) and the implementations.

                            In fact, I do want a different standard: one that is closer to my idea what the rules of the language should be in order to make the language useful, beautiful, and closer to the spirit of the design.

                            Ah, the Joys of Standards! They are so useful, everybody wants their own one! ;-)

                            Except for bit-fields, objects are composed of contiguous sequences of one or more bytes, the number, order, and encoding of which are either explicitly specified or implementation-defined.”

                            Umm. So, yes, an array is a contiguous sequence, but we’re talking about indexing out of bounds of an array. So what is contiguous beyond that array?

                            Answer 1 : Possibly empty padding to align the next object at the appropriate alignment boundary.

                            Answer 2: Which is the next object? That is determined by the field order within a struct…. (with the alignment padding determined by the ABI), but if the array is not in a struct…. it’s all bets off as to which object the compiler linker chooses to place next.

                            Hmm. Your example didn’t format nicely (nor was it valid syntax (missing parenthesis) so let me see if I can unravel that to see what you mean…

                            struct a { 
                               int x[100];
                            }; 
                            char *b = malloc(sizeof(int)*101); 
                            struct a *y = (struct a *)b; 
                            
                            if(sizeof(struct a)) != sizeof(int)*100 ) 
                                panic(“this implementation of C won’t work for us\n”); 
                            
                            …. do stuff … 
                            
                             y->x[100] = checksum(y);
                            

                            Hmm. Not sure what you’re trying to say, but try this one….

                            #include <stdio.h>
                            #include <stdint.h>
                            
                            struct {
                                char c[5];
                                uint32_t i;
                            } s;
                            
                            uint64_t l;
                            
                            int main(void)
                            {
                               printf( "sizeof(s)=%lu\n sizeof(c)=%lu\n sizeof(i)=%lu\n", sizeof(s),sizeof(s.c),sizeof(s.i));
                               printf( "address of s=%08lx\n address of l=%08lx\n diff = %ld\n", (uintptr_t)&s, (uintptr_t)&l, ((intptr_t)&s-(intptr_t)&l));
                               return 0;
                            }
                            

                            Outputs…

                            sizeof(s)=12                                                                                                                                                                                 
                            sizeof(c)=5                                                                                                                                                                                 
                            sizeof(i)=4                                                                                                                                                                                 
                            address of s=00601050                                                                                                                                                                        
                            address of l=00601048                                                                                                                                                                       
                            diff = 8                                                                                                                                                                                    
                            

                            https://onlinegdb.com/B1Ut3031m

                            1. 3

                              In the absence of the standard, there is no “C Language”, merely a collection of competing implementations of different languages, confusingly all named “C”.

                              What a weird idea. So much of the core code of the internet and infrastructure was written in an impossible language prior to the first ANSI standard. And it even worked!

                              Ah, the Joys of Standards! They are so useful, everybody wants their own one! ;-)

                              There is a standards process. It involves people presenting proposals for modifications and discussing their merits. There are changes ! That’s totally normal and expected.

                              The moment C compilers began padding, every compiler added a “packed” attribute. The reason is that many C applications require that capability. Imagine ethernet packets with artisanal compiler innovative ordering. And those attributes are not in the standard - yet they exist all the same.

                              Your example is not my example.

                              1. 2

                                So much of the core code of the internet and infrastructure was written in an impossible language prior to the first ANSI standard. And it even worked!

                                Were actually written in whatever dialect was available on the day and worked for that machine, that compiler on that day.

                                And porting to a different machine, different compiler, different version of the compiler, was a huge pain in the ass.

                                I know.

                                I’ve done a hell of a lot of that over the decades.

                                Role on tighter standards please.

                                Yup, and most of them added a subtly different packed attribute, and since it was not terribly well documented and defined, I’ve had a fair amount of pain from libraries written (LWIP comes to mind), where at various points in their history got the packed attribute wrong, so it wasn’t portable from 32 to 64 bit.

                                Your example is not my example.

                                You example wasn’t formatted, and I didn’t quite understand the point of it. Can you format it properly and expand a bit on what you were sayign with that example?

                    1. 9

                      It’s a neat language that I hope to see more of. That said, I haven’t seen any evidence of this:

                      “but the argument is that the increase in productivity and reduction of friction when memory-safe mechanisms are absent more than make up for the time lost in tracking down errors, especially when good programmers tend to produce relatively few errors.”

                      I have seen studies show safe, high-level programming give productivity boosts. I’ve also seen a few that let programmers drop into unsafe, manual control where they want with that wrapped in a safe or high-level way. Combining these should probably be the default approach unless one can justify benefits of not doing so. Also, we’ve seen some nice case studies recently of game developers getting positive results trying Design-by-Contract and Rust. If author was right, such things would’ve slowed them down with no benefits. Similarly for the game developers that are using Nim with its controllable, low-latency GC.

                      1. 4

                        It’s not mentioned in the primer but the compiler has built in support for linting. You can access to the AST during compilation and stop the build, so rather than enforcing good practice by shoehorning it into types you can enforce good practice by directly checking for misuse.

                        I do wonder if people will just end up badly implementing their own type systems on top of that though.

                        1. 2

                          That’s exactly the kind of thing I was describing in my response to akkartik. Neat stuff.

                        2. 1

                          You’re making some big claims here (even a normative statement), seemingly without much to back it up.

                          I have seen studies show safe, high-level programming give productivity boosts.

                          Too general. For some tasks high-level languages are clearly desirable, for others they clearly don’t work at all. The question at hand is which level of abstraction is desirable for writing/maintaining a large, complex codebase that has to reliable mangle a huge amount of data about 60 times per second. If a study does not replicate these conditions, it is worthless for answering the question.

                          I’ve also seen a few that let programmers drop into unsafe, manual control where they want with that wrapped in a safe or high-level way. Combining these should probably be the default approach unless one can justify benefits of not doing so.

                          Does writing your engine in C(++) and slapping Lua on top for the game logic count? Many games work like that, it pretty much is the default approach; no immature, unproven languages needed.

                          Also, we’ve seen some nice case studies recently of game developers getting positive results trying Design-by-Contract and Rust.

                          Unless a serious game in Rust has actually been released and enjoyed some success, we haven’t. A lone Rust enthusiast playing around for a few month and writing a blog post about it does not tell us a whole lot about how Rust might fare under realistic conditions.

                          If author was right, such things would’ve slowed them down with no benefits.

                          What makes you so sure they haven’t?

                          Every time I read posts like yours I wonder if Rust evangelists and other C haters ever seriously ponder why people use and like C, why it became the giant on whose shoulders so much of modern software stands. I’m not claiming it all comes down to technical superiority, but I do think there is a reason C has stood the test of time like no other programming language in computing’s (admittedly not very long) history. And it certainly wasn’t for a lack of competition.

                          Edit: I was reminded of Some Were Meant for C, which should be required reading for anyone developing a new “systems” language.

                          1. 3

                            “Every time I read posts like yours I wonder if Rust evangelists and other C haters ever seriously ponder why people use and like C, why it became the giant on whose shoulders so much of modern software stands.”

                            It’s rare I get accused of not reading enough history or CompSci papers on programming. I post them here constantly. On the contrary, it seems most C programmers didn’t study the history or design of the language since most can’t even correctly answer “what language invented the philosophy of being relatively small, efficiency focused, and the ‘programmer is in control’ allowing direct manipulation of memory?” They’ll almost always say C sine their beliefs come from stories people told them instead of historical research. Correct answer was BCPL: the predecessor that C lifted its key traits off of. Both were designed almost exclusively due to extremely-limited hardware of the time. They included just what could run and compile on weak hardware modified with arbitrary, personal preferences of the creators. Here’s a summary of a presentation that goes paper by paper studying the evolution of those designs. The presentation is Vimeo link in references. Only thing I retract so far is “not designed for portability” since a C fan got me a reference where they claimed that goal after the fact.

                            “You’re making some big claims here (even a normative statement), seemingly without much to back it up.”

                            After accusing me of doing no research, you ironically don’t know about the prior studies comparing languages on traits like these. There were many studies done in research literature. Despite the variance in them, certain things were consistent. Of the high-level languages, they all killed C on productivity, defect rate, and maintenance. Ada usually beat C++, too, except in one study I saw. Courtesy of Derek Jones, my favorite C vs Ada study is this since it’s so apples to apples. Here’s one from patrickdlogan where they do same system in 10 languages with everything that’s been mainstream beating C in productivity. Smalltalk leaves all them in the dust. The LISP studies showed similar boost in development speed. Both supporting quick iterations, working with live image/data, and great debugging help with that. LISP also has its full-strength, easy-to-run macros with their benefits. Example of benefits of LISP-based OS and Smalltalk live-editing (search for “Apple” for part about Jobs’ visit).

                            Hell, I don’t have to speculate to you given two people implemented C/C++ subset in a Scheme in relatively short time maintaining the benefits of both. Between that, Chez Scheme running on a Z80, and PreSceme alternative to C, that kind of shows that we’ve been able to do even what C does with more productivity, safer-by-default, easier debugging, and easier portability since somewhere from the 1980’s to mid-to-late-1990’s depending on what you desired. If a modern take, one could do something like ZL above in Racket Scheme or IDE-supported Common LISP. You’d get faster iterations and more productivity, esp extensibility, due to better design for these things. Good luck doing it with C’s macros and semantics. That combo is so difficult it just got a formal specification w/ undefined behavior a few years ago (KCC) despite folks wanting to do that for decades. The small LISP’s and Pascals had that stuff in the 1980-1990’s due to being designed for easier analysis. Other tech have been beating it on predictable concurrency such as Concurrent Pascal, Ada Ravenscar, Eiffel SCOOP, and recently Rust. Various DSL’s and parallel languages do easier parallel code. They could’ve been a DSL in the better-designed C which is approach some projects in Rust and Nim are taking due to macro support.

                            So, we’re not knocking it over speculative weaknesses. It was a great bit of hackery back in the early 1970’s for getting work done on a machine that was a piece of shit using borrowed tech from a machine that was a bigger piece of shit. Hundreds of person-years of investment into compiler and processor optimizations have kept it pretty fast, too. None of this changes that it has bad design relative to competitors in many ways. None of it changes that C got smashed every time it went against another language balancing productivity, defect rate, and ease of non-breaking changes. The picture is worse when you realize that people building that ecosystem could’ve built a better language that FFI’d and compiled to C as an option to get its ecosystem benefits without its problems. That means what preserves C is basically a mix of inertia, social, and economic factors having about nothing to do with its “design.” And if I wanted performance, I’d look into synthesis, superoptimizers, and DSL’s designed to take advantage of modern, semi-parallel hardware. C fails on that, too, these days that most people code that stuff in assembly by hand. Those other things have recently been outperforming them, too, in some areas.

                            There’s not a technical reason left for its superiority. It’s just social and economic factors at this point driving the use of A Bad Language for today’s needs. Better to build, use, and/or improve A Good Language using C only where you absolutely have to. Or want to if you simply like it for personal preference.

                            @nullp0tr @friendlysock I responded here since it was original article that led to re-post of the other one. I’d just be reusing a lot of same points. I also countered the other article in the original submission akkartik linked to in that thread.

                            1. 3

                              It’s rare I get accused of not reading enough history or CompSci papers on programming.

                              And I didn’t. I’m aware of your posts and interests. Rather, I’m accusing you of reading too much and doing too little. In the real world, C dominates. Yet studies have found only disadvantages. Clearly the studies are missing something fundamental. Instead of looking for that missing something, you keep pointing at the studies, as if they contain the answer. They cannot, because their conclusions fly in the face of reality.

                              After accusing me of doing no research, you ironically don’t know about the prior studies comparing languages on traits like these.

                              I’m aware of (some of) the research. Color me unimpressed. I don’t need a study to know that C is a crappy language for application development. Nor to figure out that C has serious warts and problems all over, even for OS development.

                              What I would very much like to see more research on is why C keeps winning anyway. You chalk it up to “socio-economic factors” and call it a day. I call that a job not finished. It doesn’t explain why people use and like products built in C. It doesn’t explain why some very capable engineers defend C with a vigor that puts any Rust evangelist to shame[1]. It doesn’t explain what these socio-economic factors are, but they clearly matter a great deal. Otherwise, we agree, C wouldn’t be where it is today.[2]

                              Legacy/being well-established is certainly one factor keeping C alive, but how did it become so dominant? As a computer history buff you might argue that it is because universities and their inhabitants had cheap access to UNIX systems back in the day and so students learned C and stuck to it when they graduated. There’s a lesson in that argument: having experience in a programming language is a very important factor in how productive you will be in it, which means throwing that experience away by using a very different language is a huge loss. Those who earn their bread and butter by having and using that experience will be very reluctant to do so. Most working programmers simply can/will not afford to learn a radically different language. It is an even bigger loss to society at large than to individuals because you loose teachers, senior engineers and others who pass on experience too. There are other network/feedback effects (ignoring the obvious technical ones), but I think you get the point.

                              The consequence is that small incremental improvements on existing (and proven) technology are vastly preferable to top-down fundamental redesigns, even if that means the end result isn’t anywhere close to pretty. This has been proven again and again across industries and live in general. x86 is another prominent example.

                              And then there is Go. Here’s a language that is built by people who understand C and the need to ease adoption. And look! People are actually using it! Go has easily replaced more C code on my machines than all the strongly typed, super-duper safe languages combined and probably in the internet-at-large too. And, predictably, the Haskell enthusiasts rage and shake their fists at it, because it isn’t the panacea they imagine their favorite strongly-typed language to be. Or in other words: because they do not understand, and are unwilling to learn, the most basic aspects about what keeps C alive.

                              Anyway, I wanted to circle back to your original claim that memory-safety mechanisms do not inhibit productivity in game programming, but this is already pretty long and I’m hungry. Maybe I’ll write another comment to address that later.

                              [1] And they make good on their claims by building things that are actually successful, instead of sticking to cheap talk about the theoretical advantages and showing off their skills at doing weird stuff to type systems.

                              [2] That does not mean I don’t believe there are important technical aspects to C which are missing in languages intended to replace it. See the “Some were meant for C” paper.

                              1. 0

                                “What I would very much like to see more research on is why C keeps winning anyway.”

                                It’s not “winning anyway” any more than Windows keeps winning on the desktop, .NET/Java in enterprise, PHP in web, and so on. It got the first mover advantage in its niche. It spread like wildfire via UNIX and other platforms that got huge. It was a mix of availability, hacker culture, and eventually open-source movement. Microsoft and IBM were using monopoly tactics suing or acquiring competitors that copied their stuff with different implementations or challenged them. Even companies like Borland with Pascals outperforming C on balance of productivity and speed saw the writing on the wall. Jumped on the bandwagon adding to it. The momentum and effects of several massive trends moving together who all found common ground in C at that time led to oligopolies of legacy code and decades of data locked into it. They have massive, locked-in base of code exposing C and C++. To work on it or add to it, the easiest route was learning C or C++ which increased the pool of developers even more. It’s self-reinforcing after a certain point.

                                I don’t have hard data on why non-UNIX groups made their choices. I know Apple had Pascal and LISP at one point. They eventually went with Objective-C for some reason with probably some C in there in the middle. IBM was using PL/S for some mainframe stuff and C for some other stuff. There’s gaps in what I can explain but each one I see is a company or group jumping on it. Then, it gets more talent, money, and compilers down the line. What explains it is Gabriel’s Worse is Better approach, herd mentality of crowds, network effects, and so on. You can’t replicate the success of C/C++ as is since it appeared at key moments in history that intersected. They won’t repeat. There will be similar opportunities where something outside a language is about to make waves where a new language can itself make waves by being central to or parasite off it. Economics, herd mentality, and network effects tell us it will work even better if it’s built on top of an existing ecosystem. A nice example is Clojure that built something pretty different on top of Java’s ecosystem. Plus all the stuff allowing seemless interoperability with C, building on Javascript in browsers, or for scripting integration with something like Apache/nginx.

                                So, it’s pretty clear from the lack of design to the enormous success/innovation to the stagnation of piles of code that C’s gone through some cycle of adoption driven by economics. There’s lasting lessons that successful alternatives are using. Once you know that, though, there’s not much more to learn out of necessity. Plenty for curiosity but the technical state of the art has far advanced. That’s about the best I can tell you about that part of your post after a long day. :)

                                “There’s a lesson in that argument: having experience in a programming language is a very important factor in how productive you will be in it, which means throwing that experience away by using a very different language is a huge loss.”

                                Now we’re in the “C is crap with certain benefits from its history that might still justify adoption” territory. This is essentially an economic argument saying you’ve invested time in something that will pay dividends or you don’t want to invest or have time to get a new language to that point. You see, if it’s designed right, this isn’t going to hurt you that much. The safe alternatives to C in language and libraries kept all that investment while making some stuff safe-by-default. There’s some languages that are just easy to understand but compile to C. Then, there’s those that are really different that still challenge your claim. The apples-to-apples study I linked on Ada and C specifically was concerned about their C developers doing worse in Ada due to a lack of experience. However, those developers did better since the language basically stopped a lot of problems that even experienced developers kept hitting in C since it couldn’t catch them. At least for that case, this concern was refuted even with a safe, systems language with a steep learning curve.

                                So, it’s not always true. It is worth considering carefully, though. Do note that my current advice for C alternatives is keeping close enough to C to capitalize on existing understanding, code, and compilers.

                                “Most working programmers simply can/will not afford to learn a radically different language.”

                                That’s a hypothesis that’s not proven. If anything, I think the evidence is strongly against you with programmers learning new languages and frameworks all the time to keep their skills relevant. Many are responding positively to the new groups of productive, safe languages such as Go, Rust, and Nim. Those fighting the borrow-checker in Rust about to quit usually chill when I tell them they can just use reference counting on the hard stuff till they figure it out. It’s a fast, safe alternative to Go at that point. If they want, they can also turn off the safety to basically make it a high-level, C alternative. There’s knobs to turn to reduce difficulty of using new, possibly-better tooling.

                                “People are actually using it! “

                                Go was a language designed by a multi-billion dollar corporation whose team had famous people. The corporation then pushed it strongly. Then, there was adoption. This model previously gave us humongous ecosystems for Java and then C#/.NET. They even gave Java a C-like syntax to increase adoption rate. Go also ironically was based on the simple, safe, GC’d approach of Niklaus Wirth that Pike experienced in Oberon-2. That was a language philosophy C users fought up to that point. Took a celebrity, a big company, and strong focus on tooling to get people to try what was essentially a C-like Oberon or ALGOL68. So, lasting lessons are getting famous people involved, have big companies with platform monopolies pushing it, make it look kind of like things they like, and strong investments in tooling as always.

                                “Anyway, I wanted to circle back to your original claim that memory-safety mechanisms do not inhibit productivity in game programming, but this is already pretty long and I’m hungry. Maybe I’ll write another comment to address that later.”

                                I’m interested in that. Remember, though, that I’m fine with being practical in an environment where high, resource efficiency takes priority over everything else. In that case, the approach would be safe-by-default with contracts for stuff like range checks or other preconditions. Tests generated from those contracts with automatic checks showing you the failures. If performance testing showed it too slow, then the checks in the fast path can be removed where necessary to speed it up. Throw more automated analysis, testing, or fuzzing at those parts to make up for it. If it’s a GC, there’s low-latency and real-time designs one might consider. My limited experience doing games a long time ago taught me memory pools help in some places. Hell, regions used in some safe languages sound a lot like them.

                                So, I’m advocating you use what safety you can by default dialing it down only as necessary to meet your critical requirements. The end result might be way better than, slightly better than, or equivalent to C. It might be worse if it’s some combo of a DSL and assembly for hardware acceleration which simply can have bugs just due to immaturity. Your very own example of Go plus recent ones with Rust show a lot of high-performance, latency-sensitive apps can go safe by default picking unsafety carefully.

                        1. 2

                          The netcode.io spec talks about crypto but it seems strange to me, using pubkey crypto for messages instead of doing a key exchange.

                          Valve also recently published their netcode library. It’s not documented but does link to the QUIC crypto spec which might be worth reading.

                          1. 43

                            bing has no qualms hitting meta more than 5000 times in a 3 hour period

                            We decided to take this drastic measure to protect Discourse sites out there from being attacked by Microsoft crawlers.

                            One read every two seconds is not an attack, it’s totally irrelevant unless your software is many (at least 3?) orders of magnitude too slow.

                            Maybe they should work on that before shifting the blame onto Microsoft.

                            1. 10

                              In C++ this is

                              template< typename T >
                              constexpr T max( T a, T b ) { return a > b ? a : b; }
                              

                              I know Linus is afraid of the typical/“modern” C++ crowd killing his codebase, but you don’t have to write C++ like that. You can write your normal C, just with templates/constexpr functions instead of junky macro hacks like this.

                              1. 3

                                I was thinking the same thing.

                                There are plenty of arguments against C++, but it’s silly to rage against C++ and then allow code like this.

                              1. 3

                                I suspect the hash function he uses is not ideal.

                                Multiplying integers is really surprisingly slow. On Skylake, you can start one r32 IMUL per cycle and you have to wait 4 cycles for the result. You can start four AND/OR/XORs per cycle with 1 cycle latency, so that’s 16 bitops in the same time as a single multiply.

                                1. 4

                                  The multiplier is 31, which is a sign the hash function is designed to be implemented by bitshifting. That is, h * 31 is the same as (h << 5) - h. I use a similar DJB hash here:

                                  https://github.com/google/rappor/blob/master/analysis/cpp/find_cliques.cc#L81

                                  In that case the multiplier is 33, so you do (h << 5) + h.

                                  I think Java has problems with shifts because it doesn’t have unsigned integers? But the JVM is pretty performance-minded so I imagine it should do “the right thing”.

                                1. 6

                                  I ported my compressor to use AVX2. Most of it was find and replace because most SSE instructions have a same-but-twice-as-wide partner in AVX. Not all of it was that easy though, SSE has some instructions that work on the entire 128bit register (e.g. PSHUFB) while their AVX counterparts work on 2x128bits rather than 1x256bits. Took some staring to figure out which parts were actually broken and the fixes are ugly and slow but it works now.

                                  The AVX implementation is somewhat faster than the SSE implementation, which I’m pretty sure makes it the fastest decoder in the world. Compression is not awesome but I have some experiments to try which may also end up making it faster as a bonus.

                                  I’ve been having a lot of fun working on this and I’ve been thinking about how to move my career towards this kind of work too. I like algorithms and low level performance tweaking with more science than engineering. It seems to me that jobs like this simply do not exist (obviously they do but they’re extremely rare) so I’m investigating starting my own company to work on it. Going to try to schedule some phone calls with people that have taken similar paths and other people who might be interested in buying it and see what happens from there.

                                  1. 4

                                    Garbage in, garbage out. If you run the same test with

                                    template< typename A, typename B >
                                    struct Pair {
                                        A a;
                                        B b;
                                    };
                                    

                                    in place of std::pair, you get

                                    pair.cc: In function ‘int main()’:
                                    pair.cc:14:28: error: use of deleted function ‘Pair<std::basic_string<char>, A>::Pair()’
                                         Pair< std::string, A > p;
                                                                ^
                                    pair.cc:8:8: note: ‘Pair<std::basic_string<char>, A>::Pair()’ is implicitly deleted because the default definition would be ill-formed:
                                     struct Pair {
                                            ^~~~
                                    pair.cc:8:8: error: no matching function for call to ‘A::A()’
                                    pair.cc:4:5: note: candidate: A::A(int)
                                         A( int ) { }
                                         ^
                                    pair.cc:4:5: note:   candidate expects 1 argument, 0 provided
                                    pair.cc:3:8: note: candidate: constexpr A::A(const A&)
                                     struct A {
                                            ^
                                    pair.cc:3:8: note:   candidate expects 1 argument, 0 provided
                                    pair.cc:3:8: note: candidate: constexpr A::A(A&&)
                                    pair.cc:3:8: note:   candidate expects 1 argument, 0 provided
                                    

                                    This is a problem with the STL, not gcc.

                                    1. 12

                                      At home, I’ve been optimising my compression algorithm.

                                      The encoder hasn’t received much attention yet, but I found a very nice trick for optimising one small part and it doubled the speed of the whole thing, putting it on par with LZ4. With tweaking it should beat LZ4 quite handily.

                                      The decoder is where I’ve been focusing my efforts, and it destroys anything in the open source world. I’ve seen 60% faster with the same or a bit better compression than LZ4 on normal non-degenerate files, and I still have work to do. There’s definitely room for tweaking, and I want to try widening to AVX2. AVX is a bit derp though so I’m not sure if that will work out.

                                      This is all particularly exciting because LZ4 has the world’s fastest practical LZ encoder (ruling out things like density) and I believe I’m in the same ballpark as the world’s fastest decoder (but with much less compression :<).

                                      Next steps are to become #1 and figure out how to turn it into money.

                                      1. 10

                                        I believe there’s a ‘documentary’ called Silicon Valley that can be a good starting guide.

                                        1. 2

                                          I think they even had a real-world, case study toward the end involving life-like hands and cylindrical objects. Might be useful for bicycle games or something.

                                        2. 2

                                          Good luck with your project!

                                          I’m observing compression-related subjects from the sidelines occasionally. (Lately I started providing some moderately recent Windows binaries for some compression-related tools, that are not widely available.)

                                          Are you perhaps unaware of encode.ru forum? You can look there for some projects that were based on LZ4, like LZ4X or LZ5, and a lot of other interesting stuff, like promising RAZOR - strong LZ-based archiver for instance. You’ll find there knowledgeable people from the field and their insightful posts.

                                          1. 1

                                            Giving RAD a run for their money eh? :)

                                            1. 1

                                              Sort of. I’m trying to avoid taking them head on because I won’t win that. AFAIK they don’t put much effort into having super fast encoders, which leaves room for me to focus on end to end latency.

                                          1. 2

                                            This seems like spam to me?

                                            They’ve copy pasted the advisory and put a very stupid title on it. “Ugly, perfect ten-rated bug”, “patch before they’re utterly p0wned”, “btw it’s a denial of service attack”.

                                            No thanks.

                                            1. 2

                                              It’s The Register; the self-admitted British tabloid (like the Daily Mail) of the IT world. Sometimes they can produce a good article, othertimes it’s clickbait where you’re also expecting page 3 to be a naked woman.

                                              1. 1

                                                This vulnerability can “allow the attacker to execute arbitrary code and obtain full control of the system,” it’s not just a DoS.

                                                https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180129-asa1

                                              1. 4

                                                On my PC at least, the process name has parens around it, so you can parse it with the Lua pattern %b(). Fishing out the parent PID would be like %S+ %b() %S+ (%S+).

                                                I don’t think the problem here is that /proc/pid/stat is broken, rather the string parsing tools in libc are not good enough and shouldn’t be used.

                                                The OpenBSD guys already ripped the patterns code out of Lua for their web server (patterns.c/.h), so it’s simple to add it to your own code base.

                                                1. 9

                                                  Assuming the rules for process names is the same as the rules for file names, a process name can contain unbalanced parens. What if a process is named for example hello) world? The string in /proc//stat would be (hello) world), and %b() would just find (hello). This doesn’t seem like something which could be fixed by smarter parsing, other than by reading from the end.

                                                  1. 2

                                                    Oh good point, %((.*)%) works, assuming they don’t add parens to any other fields.

                                                  2. 2

                                                    As noted in the original post, even using the parentheses does not guard against this. The only thing I can think of to safely use the current format is scanning the whole file into a buffer and searching for the last closing parenthesis, then taking everything from the first opening parenthesis to the last as executable name.

                                                    This is also not specific to C or libc, this format is bad to parse (and the mistake easy to make) with any language.

                                                  1. 3

                                                    This looks racy to me, can someone explain where I’m going wrong?

                                                    Thread A is the first to acquire the benephore, checks and increments the atomic variable, finds it is zero and proceeds without acquiring the semaphore.

                                                    While Thread A is in the critical section Thread B arrives and acquires the benephore, finds the atomic variable to be non-zero so acquires the semaphore. Because it has the semaphore it proceeds into the critical section. Now there are two threads concurrently in the critical section.

                                                    What prevents this scenario?

                                                    1. 3

                                                      I think you’re right, unless I’m missing something obvious.

                                                      Worse still, if T1 enters the critical section and is followed by T2, if T1 now makes progress it will find benaphore_atom > 1 and call release_sem() on a semaphore it doesn’t hold. Which is probably either a crash or UB, who knows.

                                                      I was missing something obvious.

                                                      The semaphore, is initialized into an ‘unavailable’ state.

                                                      When Thread B attempts to acquire the newly initialized semaphore, it blocks as the semaphore is in its ‘unavailable’ state. Thread A later finishes up in its Critical Section, and seeing that benaphore_atom > 1 it increments the semaphore, allowing Thread B to make progress.

                                                      At the end of this execution, T2 sees !(benaphore_atom > 1) and continues without marking the semaphore as available.

                                                      1. 1

                                                        Semaphores don’t provide mutual exclusion.

                                                        You use them to e.g. count the number of items in a queue and wake up enough threads to process them. Then those threads use a separate mutex to synchronise any operations on the queue.

                                                      1. 5

                                                        If you’re benchmarking a really small piece of code in a loop, you also need to be aware that

                                                        1. you can spend 50% of the time per iteration on loop overhead if you don’t unroll and
                                                        2. CPUs execute multiple instructions in parallel, so if you aren’t careful measuring the time to do 1000 iterations and dividing by 1000 will underestimate the time to run one lone iteration (throughput vs latency)
                                                        1. 6

                                                          I’m starting work on proper FPS networking for my game. A lot of the concepts have clicked in my head lately so I feel ready to work on it, but it’s so crazy hard. There’s tons of tricky stuff I need to implement and when I sit down to start on it I just have no idea what to type. It feels like I’m programming for the first time again.

                                                          So for the moment I’m doing mostly random exploration and hoping I find something reasonable. Wish me luck!

                                                          1. 30

                                                            All of them:

                                                            The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                            All of them:

                                                            The fact that they waste so much effort on incremental builds when the compilers should really be so fast that you don’t need them. You should never have to make clean because it miscompiled, and the easiest way to achieve that is to build everything every time. But our compilers are way too slow for that.

                                                            Virtually all of them:

                                                            The build systems that do incremental builds almost universally get them wrong.

                                                            If I start on branch A, check out branch B, then switch back to branch A, none of my files have changed, so none of them should be rebuilt. Most build systems look at file modified times and rebuild half the codebase at this point.

                                                            Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                                                            Virtually all of them:

                                                            Making me write a build spec in something that isn’t a normal good programming language. The build logic for my game looks like this:

                                                            if we're on Windows, build the server and all the libraries it needs
                                                            if we're on OpenBSD, don't build anything else
                                                            build the game and all the libraries it needs
                                                            if this is a release build, exit
                                                            build experimental binaries and the asset compiler
                                                            if this PC has the release signing key, build the sign tool
                                                            

                                                            with debug/asan/optdebug/release builds all going in separate folders. Most build systems need insane contortions to express something like that, if they can do it at all,

                                                            My build system is a Lua script that outputs a Makefile (and could easily output a ninja/vcxproj/etc). The control flow looks exactly like what I just described.

                                                            1. 15

                                                              The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                              I disagree. Making the build system part of the language takes away too much flexibility. Consider the build systems in XCode, plain Makefiles, CMake, MSVC++, etc. Which one is the correct one to standardize on? None of them because they’re all targeting different use cases.

                                                              Keeping the build system separate also decouples it from the language, and allows projects using multiple languages to be built with a single build system. It also allows the build system to be swapped out for a better one.

                                                              Codebases easily fit in RAM …

                                                              Yours might, but many don’t and even if most do now, there’s a very good chance they didn’t when the projects started years and years ago.

                                                              Making me write a build spec in something that isn’t a normal good programming language.

                                                              It depends on what you mean by “normal good programming language”. Scons uses Python, and there’s nothing stopping you from using it. I personally don’t mind the syntax of Makefiles, but it really boils down to personal preference.

                                                              1. 2

                                                                Minor comment is that the codebase doesn’t need to fit into ram for you to hash it. You only need to store the current state of the hash function and can handle files X bytes at a time.

                                                              2. 14

                                                                When I looked at this thread, I promised myself “don’t talk about Nix” but here I am, talking about Nix.

                                                                Nix puts no effort in to incremental builds. In fact, it doesn’t support them at all! Nix uses the hashing mechanism you described and a not terrible language to describe build steps.

                                                                1. 11

                                                                  The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                                                                  I’m not sure if I would agree with this. Wouldn’t it just make compilers more complex, bigger and error prone (“anti-unix”, if one may)? I mean, in some cases I do appriciate it, like with go’s model of go build, go get, go fmt, … but I wouldn’t mind if I had to use a build system either. My main issue is the apparent nonstandard-ness between for example go’s build system and rust’s via cargo (it might be similar, I haven’t really ever used rust). I would want to be able to expect similar, if not the same structure, for the same commands, but this isn’t necessarily given if every compiler reimplements the same stuff all over again.

                                                                  Who knows, maybe you’re right and the actual goal should be create a common compiler system, that interfaces to particular language definitions (isn’t LLVM something like this?), so that one can type compile prog.go, compile prog.c and compile prog.rs and know to expect the same structure. Would certainly make it easier to create new languages…

                                                                  1. 2

                                                                    I can’t say what the parent meant, but my thought is that a blessed way to lay things out and build should ship with the primary tooling for the language, but should be implemented and designed with extensibility/reusability in mind, so that you can build new tools on top of it.

                                                                    The idea that compilation shouldn’t be a special snowflake process for each language is also good. It’s a big problem space, and there may well not be one solution that works for every language (compare javascript to just about anything else out there), but the amount of duplication is staggering.

                                                                    1. 1

                                                                      Considering how big compilers/stdlibs are already, adding a build system on top would not make that much of a difference.

                                                                      The big win is that you can download any piece of software and build it, or download a library and just add it to your codebase. Compare with C/C++ where adding a library is often more difficult than writing the code yourself, because you have to figure out their (often insane) build system and integrate it with your own, or figure it out then ditch it and replace it with yours

                                                                    2. 8

                                                                      +1 to all of these, but especially the point about the annoyance of having to learn and use another, usually ad-hoc programming language, to define the build system. That’s the thing I dislike the most about things like CMake: anything even mildly complex ends up becoming a disaster of having to deal with the messy, poorly-documented CMake language.

                                                                      1. 3

                                                                        Incremental build support goes hand in hand with things like caching type information, extremely useful for IDE support.

                                                                        I still think we can get way better at speeding up compilation times (even if there’s always the edge cases), but incremental builds are a decent target to making compilation a bit more durable in my opinion.

                                                                        Function hashing is also just part of the story, since you have things like inlining in C and languages like Python allow for order-dependent behavior that goes beyond code equality. Though I really think we can do way better on this point.

                                                                        A bit ironically, a sort of unified incremental build protocol would let compilers avoid incremental builds and allow for build systems to handle it instead.

                                                                        1. 2

                                                                          I have been compiling Chromium a lot lately. That’s 77000 mostly C++ (and a few C) files. I can’t imagine going through all those files and hashing them would be fast. Recompiling everything any time anything changes would probably also be way too slow, even if Clang was fast and didn’t compile three files per second average.

                                                                          1. 4

                                                                            Hashing file contents should be disk-io-bound; a couple of seconds, at most.

                                                                            1. 3

                                                                              You could always do a hybrid approach: do the hash check only for files that have a more-recent modified timestamp.

                                                                            2. 1

                                                                              Do you use xmake or something else? It definitely has a lot of these if cascades.

                                                                              1. 1

                                                                                It’s a plain Lua script that does host detection and converts lines like bin( "asdf", { "obj1", "obj2", ... }, { "lib1", "lib2", ... } ) into make rules.

                                                                              2. 1

                                                                                Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                                                                                Unless your build system is a daemon, it’d have to traverse the entire tree and hash every relevant file on every build. Coming back to a non-trivial codebase after the kernel stopped caching files in your codebase will waste a lot of file reads, which are typically slow on an HDD. Assuming everything is on an SSD is questionable.

                                                                              1. 11

                                                                                This is quite disappointing, his solution for each point is basically “just get it right!”

                                                                                Here are some actual solutions for the problems he brings up:

                                                                                Not freeing memory after allocation:

                                                                                Try to avoid the “free everything individually” pattern where possible because it’s the easiest to get wrong. Use RAII if you like. Try to centralise your resource management higher up the call stack so leaf code does not call malloc/free, people coming from other languages tend to do that because it’s fine if you have a GC, but you can make big messes for yourself in C. Use easier allocation strategies like “free everything allocated after this point when this statement falls out of scope”, which works very well for temp scratch space allocations.

                                                                                All of your allocations should go through memory managers that check for leaks. The least intrusive way to do it is by keeping a hashtable that maps from the pointer to some info struct, which at least contains file and line of the allocation.

                                                                                Freeing already freed memory (double-freeing):

                                                                                Largely mitigated by what I said before. I have no specific suggestions for this because in practice I never have problems with it.

                                                                                Invalid memory access, either reading or writing:

                                                                                I don’t have any recommendations for NULL pointers, but stuffing asserts/NULL checks all over the place is a signal you have no idea what your code is doing. TBH it’s not really been a problem for me.

                                                                                For uninitialised data, making all your allocators call memset( 0 ) is a good start (try to make your classes have some valid default state when they’re all 0), beyond that it’s not really a problem because it’s pretty easy to catch. Use RAII if you like.

                                                                                Also lobby the C++ committee for something like #pragma primitive_types_initialise_to_zero_unless_you_tell_them_not_to.

                                                                                Buffer-overflow, either reading or writing:

                                                                                Implement sane primitives, such as:

                                                                                template< typename T >
                                                                                struct array {
                                                                                    T * elems;
                                                                                    size_t n;
                                                                                    T & operator[]( size_t i ) { ASSERT( i < n ); return elems[ i ]; }
                                                                                };
                                                                                

                                                                                Replace any usage of pointer + size with that class. You will probably want StaticArray< size_t, T >, DynamicArray, array2d, etc too.

                                                                                You’ll also want a sane string class, a sane string parsing library (Lua’s patterns library is very very good), and a sane stream writer/parser class. Use these classes instead of C strings/pointers + sizes.

                                                                                1. 2

                                                                                  As for your first solution, I would first check to see if the runtime can do the checks for you. I know that the GNU C library can do more checks if you set certain environment variables, and using valgrind is wonderful if you have access to it. Check first before writing your own memory wrapper (that is what lead to Heartbleed, by the way).

                                                                                  Second, pick a better value to initialize memory with than just 0. If you really want to help with debugging on the x86 platform, I suggest using the value 0xCC. As a signed quantity, it’s a large negative value that is easy to spot. As an unsigned quantity, it’s a huge number. As a character, it’s invalid ASCII (but it is a valid UTF-8 initial byte character, but two 0xCC’s in a row is invalid UTF-8). As a pointer, it’s probably an invalid pointer (so you’ll get a SIGSEGV most likely) and if it’s accidentally executed, it’s the INT3 debugging instruction.

                                                                                  I don’t really have any good solutions to string handling in C other than “don’t do that!” Which is why I use Lua if possible.

                                                                                1. 4

                                                                                  Great writeup!

                                                                                  What things didn’t work out? Any great screenshots of it just failing miserably? Any issues with normals in areas where you swap levels of detail?

                                                                                  1. 4

                                                                                    The main thing that didn’t work was trying to draw just the square tiles and nothing else. The failures from that aren’t very exciting, you end up with overlap on one side and a gap on the other as the inner clipmap levels move around.

                                                                                    The funniest bugs I ran in to were all to do with the snapping. If you don’t snap the levels at all it looks like this, at one point I had a bug where I was snapping the texture coordinates but not the actual world position of the mesh so the world would tremble as the camera moved around, and my best bug had the entire terrain flapping up and down like mad. (don’t remember how I did that one sadly)

                                                                                    Also lots of mundane garbage with mesh generation, like random triangles stretching across the whole world and holes in the ground.

                                                                                    Normals are mostly fine across transitions, but if there’s high frequency detail in the terrain there can be quite a big pop.

                                                                                  1. 16

                                                                                    I really can’t feel bad for companies on this. They’ve demonstrated over and over again that they can’t be trusted to do the right thing on their own.

                                                                                    I wish the United States could enact a law like this, but who am I kidding?

                                                                                    1. 0

                                                                                      I really can’t feel bad for companies on this. They’ve demonstrated over and over again that they can’t be trusted to do the right thing on their own.

                                                                                      Even the one-man companies currently just starting out that are under threat of 20M EUR fines for not complying with rules that are practically impossible to comply with?

                                                                                      You’re not seeing the big picture here. The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can.. It’s fucking ridiculous.

                                                                                      1. 18

                                                                                        Even the one-man companies currently just starting out that are under threat of 20M EUR fines for not complying with rules that are practically impossible to comply with?

                                                                                        Especially those ones because otherwise they have no checks and balances whatsoever and the single person in charge will do whatever they feel like without telling anybody.

                                                                                        We don’t let small restaurants ignore food safety, or small construction companies ignore building codes, why would we let small internet companies ignore privacy regulations?

                                                                                        You’re not seeing the big picture here. The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can.. It’s fucking ridiculous.

                                                                                        Just because the government is spying on us doesn’t mean we should allow corporations to do it too. We don’t have to solve both problems at the same time.

                                                                                        1. [Comment from banned user removed]

                                                                                          1. 7

                                                                                            I’ve downvoted you as troll because we don’t have anything like “unnecessarily rude”.

                                                                                            Please try to be less abrasive with your posts.

                                                                                            1. 5

                                                                                              Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?

                                                                                              Another way of putting this is “do you sincerely think that small companies need to be threatened with being put out of business entirely if they disregard their customers’ safety?” and my answer is yes, absolutely.

                                                                                              Restaurants operate perfectly well under there threat of “if you give a noticeable quantity of customers food poisoning even once, the FSA will permanently shut you down”.

                                                                                              It’s perfectly sensible to me that any other business capable of ruining a whole bunch of peoples’ lives should be held to the same standard.

                                                                                              1. 0

                                                                                                Another way of putting this is “do you sincerely think that small companies need to be threatened with being put out of business entirely if they disregard their customers’ safety?” and my answer is yes, absolutely.

                                                                                                You know you’re comparing apples to oranges, right? Someone’s e-mail address staying in a database somewhere “against his will” is not comparable to feeding people rotten/contaminated food or something.

                                                                                                Regardless, neither warrants a 20M EUR penalty, and in the first case, it’s just utterly insane. You also understand that just fine, so why are you arguing against me?

                                                                                                It’s as if EU regulators were completely oblivious to the existence of small businesses, which they’re actually not, of course, because they’re not that fucking retarded/insane.

                                                                                                So they’ve set MegaCorp level fines for everyone for some reason other than insanity/stupidity. But it’s not for no reason, and it’s not because it’s reasonable either.

                                                                                                And obvious truth is met with hostility, as always.

                                                                                                1. 2

                                                                                                  you’re comparing apples to oranges

                                                                                                  Privacy violations don’t happen at a rate of 1 or 2 per incident, they hit thousands of people at a time.

                                                                                                  neither warrants a 20M EUR penalty

                                                                                                  You won’t be paying 20M EUR. Your company will be accruing a 20M EUR debt that it will immediately fold under. You founded a limited liability company for this reason. The number could be 13.37B EUR and that outcome would be the same for small companies.

                                                                                                  Folding your company is a perfectly fair outcome for you flagrantly violating other peoples’ privacy rights.

                                                                                                  met with hostility

                                                                                                  I didn’t say anything rude to you at all. You asked a question, “Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?” and I gave a completely polite answer. You just read it as hostile because you don’t like that answer edit: because people downvoted you a lot, which, to be fair, probably feels hostile.

                                                                                              2. 2

                                                                                                Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?

                                                                                                I do sincerely think companies who can’t respect people’s privacy and don’t take the issue seriously should go out of business. The size of the company has nothing to do with it. The potential damage done once the info escapes into the wild is the same either way.

                                                                                                It’s unfortunate that it takes the threat of a 20M EUR fine and possibly going out of business to drive the point home, but asking nicely and hoping companies do the right thing hasn’t worked.

                                                                                                Who’s “we”? -The faceless bureaucracy of the EU that’s hoisting this pile of garbage on productive people?

                                                                                                If they can’t harness some of that productivity to protect the private data they collect, then good riddance.

                                                                                            2. 3

                                                                                              The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can

                                                                                              States (for better or worse) need a monopoly on coercion. Some of them have realized that the breakdown of privacy is eroding that monopoly, and they’re reacting.

                                                                                              1. [Comment from banned user removed]

                                                                                                1. 2

                                                                                                  What doesn’t make sense about it?

                                                                                                  I’m not making a value judgement about whether it’s a good thing or not; I’m giving a reason why a self-interested state would choose this course of action.

                                                                                                  Separating value judgement from behavioral reasoning is the only way to make sense of others behavior when they don’t share your values.

                                                                                              2. 3

                                                                                                Why do you think they are impossible to comply with? Germany has had laws like this for years and it works just fine.