1. 2

    The debugging technique that took me entirely too long to learn is that you can write code that helps you debug that is not “payload code” of the program you want to write. Most typically this means writing tools that interact with the payload code, for example to be able to reproduce a bug with one keypress instead of through a complex manual and error prone interaction with your system. It might also mean including non-payload logic in a program to make it easier to work with it in a debugger.

    I can’t stress enough the power of using code to debug your code. After all, if a Turing complete language is the right tool to write your payload program, why limit yourself to anything less powerful when getting your program correct?

    And then a couple of words about printing and debuggers. It didn’t take me too long to learn these two techniques, but too long to see when to use one or the other and how to combine them:

    • Printing (or logging/tracing) is really good at showing transients in your code. They show you the path to a system state, but you can’t easily explore system state without adding instrumentation everywhere. It took me too long to gain the confidence that printing is not used “by hacky programmers”, as some detractors say, but is a valuable and powerful tool for some problems.
    • A debugger visualizes the state of your program at a certain point in time. It is also good at getting to know a codebase that you are unfamiliar with. While it may be associated with “big IDEs” like VS or Java IDEs, debuggers are not “a tool for those who have no clue”, as some detractors may say, but a valuable and powerful tool for some problems.
    1. 18

      Not necessarily a technique, but I brushed off using a debugger for way too long. “if I need a debugger to understand the code then it was too complex to begin with”, “printfs are fine”, etc.

      Using a debugger with disassembly and a memory view greatly increased my productivity. It’s also very useful to learn how to use features past simple breakpoints - ie. conditional breakpoints and watch variables.

      1. 8

        A thing I see happening over and over again is that there are two camps of people: Those who only use debuggers and those who only use printing, logging, tracing etc.

        Typically the arguments of the first group are that “printf debugging” is not professional and inefficient. The latter group often says that they don’t want to learn another tool or that the need to use a debugger is a warning sign of a hard to understand code base.

        I politely disagree with both camps. The two tools have different use cases and there are areas where it is hard to operate with only one but not the other.

        In my opinion, debuggers shine for two things. The first is state visualization. Given a certain point in the execution of your program, a debugger will show you the state of the program. When I hear that a program segfaults, the state at the segfault is the first thing I want to see. A debugger will not tell me how the program got into this state*. To find that out, I will resort to reasoning, strategically adding print/log/trace statement, or, if simple, breakpoints to catch conditions before the program crashes. The second use case is working with an unfamiliar code base. Debuggers are good at answering a question like “where is this function called from?” or “is this code ever executed?” without the need to change the program.

        Prints/log/traces shine for transient visualization. Say you are debugging an FIR filter that starts to oscillate. Breaking into the code with a debugger when oscillating will probably be useless, because while now you see the system in a misbehaving state, you don’t know how it got there. Stepping might be an ineffective method, because the error conditions builds up over thousands of runs of the filter. Printing information as you go can give you a good idea how a system is moving into an undesirable state. When you are adding prints, as a bonus, you can add additional logic. For instance, you can check for certain conditions, print marking message like “here it looks like an accumulator overflowed” or print data with a compact machine representation in an easily human parsed shape that will allow you to find spurious behavior in a cursory glance.

        In summary, debuggers and prints have an overlap in their functionality, but I feel it’s most productive to combine the two in an intertwined way of working.

        • Barring reverse debugging which I can often not use due to me working with embedded systems.
        1. 6

          conditional breakpoints and watch variables

          These improved my ability to understand programs using a debugger at least ten-fold. Expression evaluation is another significant feature I use.

          1. 2

            I was very slow to pick up watchpoints. It took me fixing a bug in a debugger front-end to even understand what they were, and why they were useful.

          1. 23

            So many:

            1. If there’s a piece of code that’s not doing what you expect, add a loop around it whose condition is a variable that can be modified by the debugger. The resulting binary can be effectively unchanged such that the loop executes once, but if a debugger is present and you see the strange behavior, you can go back and step through it. 1a. This can also be done by modifying the instruction pointer, but that requires restoring register state. I do this too but it’s more tricky.
            2. The unused portion of the stack often contains clues about what happened recently.
            3. Break on access (ba in WinDbg) uses CPU registers to break on a memory address when it’s used. This is wonderful for tracking how state moves across a system, or tracking when locks are acquired and released, etc.
            4. Using breakpoints with conditions allows things like breaking on a common thing, like opening a file, printing out the state of the operation, and resuming execution, allowing interesting conditions to be logged.
            5. While very common, it’s also useful to get a debug memory allocator that can do things like mark pages as invalid on free (to generate access violations on use after free), or add a guard page after a memory allocation (to cause access violations on memory overruns.) A good one of these can also keep around allocation information after free for debugging, such as the stack that freed memory, and track the number of allocations to detect leaks.
            6. Altering thread priorities in the debugger is one way to promote the code you’re trying to look at and/or demote code that’s interfering with you.
            7. If you have a race condition, thoughtfully adding sleep in the right place can provoke it allowing it to be debugged and understood.
            8. If code aborts in ways you don’t expect, add a breakpoint to the exception filter (which decides which handlers to execute.) This executes when the full stack that caused the exception to be raised is present.
            9. Further to the previous comments about patching code, if you’re stepping through code and a branch goes somewhere you don’t want, change the instruction pointer. If you want that branch to happen again, change the condition jump to an unconditional jump.
            10. In WinDbg, dps on import tables or import table entries to show where indirect calls will go. This is useful when something else is trying to hijack your code.
            11. Keep an in memory circular log of interesting cases in your code. This log often doesn’t need to be big, and doesn’t need to allocate on insert, but if something bad happens in your code you can dump the interesting cases that were hit recently.
            1. 4

              If there’s a piece of code that’s not doing what you expect, add a loop around it whose condition is a variable that can be modified by the debugger. The resulting binary can be effectively unchanged such that the loop executes once, but if a debugger is present and you see the strange behavior, you can go back and step through it

              You don’t need this if you have a reverse debugging tool. Gdb has basic support, but I usually use rr.

              1. 4

                Reverse debugging isn’t supported universally (unfortunately) because of it’s implementation details, for example rr can’t run on a variety of apps (including the one I work on every day) because it depends on emulating syscalls, and there are a bunch which are impossible / hard to implement. See: https://github.com/mozilla/rr/issues/1053 There are also contexts where reverse debugging doesn’t work such as in kernel mode on many systems, or in other “special” environments.

                1. 1

                  Reverse debugging is incredibly useful. Unfortunately, even when using gdb, there are situations when you can not reverse debug due to lack of support from the target. This would be the case for e.g. all embedded development.

                2. 2

                  Break on access (ba in WinDbg) uses CPU registers to break on a memory address when it’s used

                  GDB / LLDB call this watchpoints. These are most useful in WinDbg if you’ve got a deterministic test failure, because you can turn on tracing, go to the crash, set the watchpoint, and then run the program in reverse.

                  If code aborts in ways you don’t expect, add a breakpoint to the exception filter

                  On *NIX, the equivalent is adding a signal handler.

                  Keep an in memory circular log of interesting cases in your code

                  I can’t overstate how useful this is. In Verona, we have a flight recorder like this (which records format strings and associated values so you can do printf-like things in the replay but don’t need to do expensive printf-like things in normal operation). This is made even more useful by systematic testing, where the runtime provides deterministic scheduling of threads, driven from a pseudo-random number generator with a provided seed. This is incredibly useful, because if we get a failure in CI then we can always reproduce it locally. I miss that facility a lot when I use any other environment.

                1. 23

                  Every single time I see a warning from python asyncio that a future wasn’t awaited, or a warning from node that a promise wasn’t used, I think that we got await backwards.

                  This feels like it’s the right way to do it. Awaiting should have been implicit, and taking a task and doing abstract work with it should have been the special case.

                  1. 9

                    It always blows my mind that it’s not a hard error to use a promise without having awaited it. Javascript will let you silently if (promise) { missiles.launch() } while all promises evaluate to true. At least Python spits out a warning, but it has the same underlying broken behavior.

                    1. 4

                      That particular error is mainly due to weakness of lack of type system and “truthy” casts rather than await-specific. Object would evaluate to true too (e.g. instead of if (authorization.canLaunchMissiles)). OTOH Rust also needs explicit .await, but won’t compile if future {} or if some_struct {}.

                      With implicit await I’d be worried about mutexes. Rust has both sync and async versions of a mutex, each with very different uses. What does “colorblind” mutex do? And how do you guarantee that code in your critical section doesn’t become automagically async, risking a deadlock?

                      1. 2

                        In Zig, the expectation will presumably be that you either support both modes or you compile-time error on the one you don’t support. Problem solved.

                        Note: it’s not clear how you would achieve the same thing in Rust, if it would even be possible.

                        1. 3

                          Rust gives you a warning at compile time if a Future is not awaited, at least. It’s actually the same infrastructure that they use to warn about unused Result values (that equates to unchecked errors).

                    2. 4

                      This is more like a co-routine based system like Go has rather than an event-based concurrency system like JavaScript.

                      1. 18

                        Yes that’s exactly right. Go + manual memory management + runtime in userland = zig evented I/O mode.

                        1. 3

                          This is perhaps explained in the video, but does switching to io_mode = .evented effectively make every function async? It’s clear that plain function calls are transparently converted to await async call(). To do so, the calling function must also be async, correct? At least, if the example given truly matches the equivalent Python given.

                          1. 5

                            It doesn’t make every function async automatically, but if you are hinting that it can propagate to many functions, yes that is an accurate observation. Any suspend point makes a function async, and await is a suspend point, as well as calling an async function without using the async keyword.

                            1. 1

                              Yes that’s what I meant, thank you!

                        2. 3

                          This is more like a co-routine based system like Go has rather than an event-based concurrency system like JavaScript.

                          These days, a lot of languages are adding some sort of async/await functionality for dealing with I/O and/or concurrency.

                          I have not understood why doing async/await is preferrable to putting focus on cheap contexts and composable communication and synchronization primitives, e.g. Go’s goroutines and channels. To me, a future that you can await, be it constructed through async or manually, looks a whole lot like a suspended thread with its associated stack. No matter whether you’re doing traditional threads or async/await, the compiler needs to generate code that manipulates the two key data structures of an independent unit of execution: a program counter to keep track where you are executing and a list of call frames where you are coming from. Threads implement this through the PC register and call frames on a stack*. Async/await implementations remember all the continuations they have, containing the equivalent information as the PC register. Continuations continue in other continuations, building the equivalent of a list of call frames where you are coming from.

                          My assumption is that threads and and async/await have the same expressive power. It also seems that they implement conceptually equivalent data structures. What is the reason for having both in a language? Am I missing something?

                          • I am not including, for instance, general purpose processor registers here. For a context switch, they can be saved to the stack if needed. Depending on ABI and cooperativity of your threading implementation, all registers might be already saved to the stack at the time of context switch anyway.
                          1. 3

                            Rust 0.x had this model, and dropped it because:

                            • coroutines must a runtime, and Rust doesn’t want to be tied to one. Rust aims to usable everywhere (including drivers, kernels, plugins, embedded hardware, etc.)

                            • coroutines need small stacks and ability to swap stacks on context switches. This adds overhead and complexity. Segmented stacks have very bad performance cliffs. Movable stacks require a GC. OTOH async/await compiled to a state machine is like a stack of guaranteed fixed size, and that stack can be heap-allocated and easily handled like any other struct.

                            • It’s incompatible with C and every other language which wants single large stack. Foreign functions will not (or don’t want to be bothered to) cooperate with the green thread runtime. This is why in Go calling out to C code has a cost, while in Rust it’s zero cost and Rust can even inline functions across languages.

                            • Rust values predictability and control. When blocking code becomes magically non-blocking there are a lot of implicit behaviors and code running behind the scenes that you don’t directly control.

                            1. 1

                              coroutines need small stacks and ability to swap stacks on context switches. This adds overhead and complexity. Segmented stacks have very bad performance cliffs. Movable stacks require a GC.

                              You don’t need small stacks but just 64-bit and overcommit. Allocate an X MiB stack for each thread and rely on the kernel to do the obvious correct thing of only committing pages you actually use.

                              Of course if you need to support 32-bit systems or badly designed operating systems this doesn’t work but it’s a very reasonable design trade-off to consider

                              1. 2

                                Fair point. That’s a clever trick, although you may need to call the kernel for every new coroutine, which makes them heavier.

                                It’s not for Rust though: it supports even 16-bit CPUs, platforms without an OS, and microcontrollers without MMU.

                                1. 1

                                  It’s a viable approach in C, yet C supports all those things too. Is the difference that Rust wants a unified approach for all platforms from 16-micros to supercomputers, unlike the fragmented land of C? If that is the case then I suppose that’s a reasonable trade-off.

                                  1. 1

                                    Yeah, Rust takes seriously ability to run without any OS, without even a memory allocator, without even Rust’s own standard library. That makes it favor solutions without any “magic”.

                                    async fn only generates structs that don’t do anything, and you have to bring your own runtime executor to actually run them. That lets people choose whether they want some clever multi-threaded runtime, or maybe just a fixed-sized buffer that runs things in a loop, or like Dropbox did, a special executor that fuzzes order of execution to catch high-level race conditions.

                      1. 18

                        Worth reading to the end just for the totally evil code snippet.

                        It was kind of foreshadowed to be evil when the author named it “skynet.c” I guess.

                        1. 4

                          Reminds me of the Java-code we used to see around 2000.

                          With a RuntimeException try-catch at the top and then just print it and continue like nothing happened.

                          How much bad bugs, data corruption and weirdness did that practice cause?

                          1. 1

                            How is that any different from kubernetes and “just restart it”? Its mostly the same practice ultimately, though with a bit more cleanup between failures.

                            1. 2

                              I guess it depends on whether you keep any app state in memory. If you’re just funnelling data to a database maybe not much difference.

                          2. 2

                            Now I start to wonder, how the correct code should look like (as opposed of jumping 10 bytes ahead).

                            Read DWARF to figure out next instruction?

                            Embed a decompiler to decode the faulty opcode length?

                            1. 4

                              Increment the instruction pointer until you end up at a valid instruction (i.e., you don’t get SIGILL), of course ;)

                              1. 6

                                I have code that does this by catching SIGILL too and bumping the instruction pointer along in response to that. https://github.com/RichardBarrell/snippets/blob/master/no_crash_kthxbai.c

                                1. 2

                                  Brilliant. I’m simultaneously horrified and amused.

                                2. 1

                                  SIGILL

                                  That’d be a pretty great nerdcore MC name.

                                3. 1

                                  If you want to skip the offending instruction, à la Visual Basics “on error resume next”, you determine instruction length by looking at the code and then increment by that.

                                  Figuring out the length requires understanding all the valid instruction formats for your CPU architecture. For some it’s almost trivial, say AVR has 16 bit instructions with very few exceptions for stuff like absolute call. For others, like x86, you need to have a fair bit of logic.

                                  I am aware that the “just increment by 1” below are intended as a joke. However I still think it’s instructive to say that incrementing blindly might lead you to start decoding at some point in the middle of an instruction. This might still be a valid instruction, especially for dense instruction set encodings. In fact, jumping into the middle of operands was sometimes used on early microcomputers to achieve compact code.

                                4. 2

                                  Here’s a more correct approach: https://git.saucisseroyale.cc/emersion/c-safe

                                  1. 1

                                    Just don’t compile it with -pg :)

                                  1. 1
                                    1. Surprising math:

                                    int x = 0xfffe+0x0001;

                                    looks like 2 hex constants, but in fact it is not.

                                    But what is it? For me that’s a syntax error and gcc agrees:

                                    error: invalid suffix "+0x0001" on integer constant
                                        2 | int x = 0xfffe+0x0001;
                                          |         ^~~~~~~~~~~~~
                                    

                                    Am I missing something?

                                    1. 2

                                      What is it? It’s a syntax error, as you determined.

                                      If you’re asking why it’s a syntax error, I believe it’s because the letter e directly followed by a plus or minus sign is used to provide the exponent of a floating-point value. It’s not legal to use this syntax on a hexadecimal constant (that’s done with the 0xfffp+1 syntax), but I’m assuming that the C standard doesn’t allow this syntax to avoid confusion.

                                      Adding a space before the plus sign makes the syntax error go away.

                                      1. 2

                                        If you’re asking why it’s a syntax error, I believe it’s because the letter e directly followed by a plus or minus sign is used to provide the exponent of a floating-point value. It’s not legal to use this syntax on a hexadecimal constant (that’s done with the 0xfffp+1 syntax), but I’m assuming that the C standard doesn’t allow this syntax to avoid confusion.

                                        That’s also my understanding of the code presented.

                                        Adding a space before the plus sign makes the syntax error go away.

                                        Yes. And then it’s also “2 hex constants” and nothing fishy is going on. I still don’t see the point of the author :D

                                      2. 1

                                        I tried this with gcc -std=c89 and it still won’t parse. I tried with DeSmet C from 1985 and it doesn’t complain, but it also doesn’t do anything surprising.

                                        C:\WOZ>type abuse.c
                                        #include <stdio.h>
                                        
                                        main()
                                        {
                                          int x = 0xfffe+0x0001;
                                          printf ("%x\n", x);
                                          return 0;
                                        }
                                        C:\WOZ>c88 abuse.c -Ic:\desmetc\
                                        C88 Compiler    V2.51    (c) Mark DeSmet, 1982,83,84,85
                                        end of C88        001F code   0004 data       1% utilization
                                        C:\WOZ>bind abuse.o
                                        Binder for C88 and ASM88     V1.92    (c) Mark DeSmet, 1982,83,84,85
                                        end of BIND        9% utilization
                                        C:\WOZ>abuse
                                        FFFF
                                        

                                        Since DOS uses 16-bit integers if you print that with %d you’ll get -1, but again, this is expected.

                                      1. 1

                                        It’s a very cool project and an interesting read.

                                        One point I would like to add: I think good performance absolutely needs predictable interrupts. For the author this works out, because his interrupt sources are timers or the VSYNC signal - both highly predictable. This becomes impossible if you simulate a wide range of systems with various interrupt sources that are not as easily specified.

                                        What I don’t understand: What happens in the case we know an interrupt is going to happen in a block? Is there a fallback to an interpreter?

                                        1. 1

                                          To mention something else: I enjoyed this project.

                                          It’s cool in a Monty Python nonsensical way. The author has answered a question that I never had - can you have a living process with no memory - with a clear and good example - that I would not have come up with.

                                          Very enjoyable content and a reason I like coming here.

                                          1. 5

                                            To me, the only alternative to adding instrumentation (i.e. logging, tracing, debug registers, test points, depending on what you work with) is making no mistakes or knowing all the emergent behaviors of your system. Keep in mind that your system can be driven in ways you did never envision.

                                            As this is impossible, I clearly prefer to have some instrumentation in the system.

                                            One part of the article that does ring a bell, though, is “fighting log levels”, as it’s called. I believe that it’s basically intractable do define a useful set of levels like e.g. CRITICAL, ERROR, EXCEPTION, FATAL, WARNING etc. For one, every system and organization has a different view on what these mean and how system behavior should be assigned to these categories. Often, there is no consensus.

                                            Secondly, and more importantly, I think that it’s typically impossible to assign a level at the point of logging. For example, consider a function that opens a file, reads the contents, closes the file and returns the content and an error code. Now imagine that you decide that opnening a nonexistant file is an error and this should be logged in the quite severe ERROR category. Imagine now that you want to use that function to read some files if they exist - if they don’t exist, no biggie, you continue. You can implemement the behavior because the error code the function returns is enough for this. However, you will now have your log sprinkled with ERRORs that are, in reality, normal and expected operation.

                                            1. 19

                                              I do not agree with the last point. Debuggers are useful and use of one does not point to poor code quality.

                                              1. 5

                                                I agree. I don’t see how using a debugger is any different from one of the suggested alternatives:

                                                Use print statements

                                                If I see code littered with echo or printf or logging statements, it says to me that you’re not sure what the code is doing there and you want some insight.

                                                  1. 2

                                                    If I see code littered with echo or printf or logging statements, it says to me that you’re not sure what the code is doing there and you want some insight.

                                                    I think this only holds if you know all the ways in which the code is driven from the outside. Many of the bugs I find, at least in my code, are the result of emergent behavior I didn’t completely foresee. They are bugs in their own right, sure, but they tend to be manifest themselves only when my code runs in the scpoe of a larger system.

                                                    In such a case I find it immensely useful to have prints, logs or traces all over the place so that I can see how the system is driven into an error state. Having the prints can also mean you’re not sure how your code is being used.

                                                  2. 5

                                                    95% of the time I have to use a debugger, it is because the code is not written in a simple and obvious way.

                                                    The other 5% of the time it’s because “some other weird thing is going on.”

                                                    Having to use a debugger to understand what code is doing is absolutely a sign of a poor code. Good code can be read on its own and understood with a very high degree of confidence.

                                                    Of course, as a programmer working with other people’s code and legacy systems, knowing how to use a debugger is important. But it still usually means you’re working with code that is overly complex.

                                                    1. 2

                                                      You can also completely understand how a piece of code works, but don’t understand why it exhibits certain behavior. This happens when your code is used in a big opaque system, and it happened to me a lot. For example:

                                                      class Person {
                                                          public Person(string name, int age) {
                                                              if (name == "" || name == null || age < 0 || age > 150)
                                                                  throw new Exception("Invalid person.");
                                                          }
                                                          
                                                          ...
                                                      }
                                                      

                                                      If the exception is thrown, it is obviously the caller’s fault. But if this exception shows up, it’s not immediately clear where something is going wrong. You would need to read through the whole code to see what’s happening.

                                                      Especially in languages with pointers or references, it is possible that a reference/pointer is copied somewhere in an unrelated function, and aliases with data you use. This way, data can be overwritten when you don’t expect it to be. I usually debug this by stepping through the code and watching the data.

                                                      … and yes, it does mean that the code is not perfect. But hey, mistakes do happen.

                                                      1. 1

                                                        The problematic code in this case – from the perspective of my original comment – is the system code.

                                                    2. 2

                                                      Debuggers are symptomatic of not understanding the code. Agree or disagree?

                                                      1. 11

                                                        Oscilloscopes are symptomatic of not understanding the circuit.

                                                        1. 2

                                                          Yes. If you understood what they were doing, you wouldn’t need instrumentation.

                                                        2. 4

                                                          Debugging is symptomatic of not understanding the code.

                                                          I think Hintjens is saying your (understanding of the) code & recent change should be so clear you don’t need a debugger, while c-jm is saying debuggers can be useful even when your code is clear and your changes were small. Both are obviously true.

                                                          1. 2

                                                            I think they are orthogonal really, code can be clean and hard to debug all at the same time.

                                                            Elements of “clean” code can also lead to making debugging harder. The more functions you have to do smaller things while more easily readable always introduces context shifts etc.

                                                            As with everything there are costs and benefits to every decision, all my point was is that in terms of a code quality the need to debug is not a good metric.

                                                          2. 1

                                                            What’s “the code” though. Abstractions layered upon abstractions. Finding bugs without debugging in legacy code or multi-threaded code is almost impossible without debugging.

                                                            1. 2

                                                              I don’t dispute that.

                                                              Yet if you indeed are able to understand your code without a debugger, that’s a very good thing.

                                                          3. 1

                                                            Often you need to debug code somebody else wrote.

                                                            1. 1

                                                              “Need” is the key part. Debuggers are super useful, but if I require them to do my job, then something is wrong.

                                                              1. 1

                                                                In my mind, debuggers are a great tool to validate and to enhance your understanding of how the code actually works. I have sympathy with the thought that it should not be necessary to run your code with a debugger to understand it. What I find weird is that the author puts this in a list together with “If your language offers a REPL that’s cool. “ which is for me in the same category of exploratory tool.

                                                              1. 8

                                                                It’s interesting, and also somewhat shocking, to hear that the largest part of programmers did not feel that programming in machine language was something that they should improve upon.

                                                                I feel very much the same way with respect to hardware design languages. I find VHDL (and Verilog, for that matter) to be insanely limited and have been looking for an alternative. Recently, I started using SpinalHDL for work purposes and cannot possibly imagine going back to the old ways. By my own estimates I am between 2-5 x more productive, depending on type of work. Some logic I would have never started writing in VHDL because opportunity cost is just too high. Most seasoned HDL engineers I tell about the language are, however, totally unimpressed. To them it seems like a risk that, even if taken, will not gain them much.

                                                                It also reminds me of the blub paradox: https://wiki.c2.com/?BlubParadox

                                                                1. 2

                                                                  It’s interesting, and also somewhat shocking, to hear that the largest part of programmers did not feel that programming in machine language was something that they should improve upon.

                                                                  Here is Edsger W. Dijkstra, in his 1972 ACM Turing Award lecture, illustrating your observation. (Emphasis added.)

                                                                  Another lesson we should have learned from the recent past is that the development of “richer” or “more powerful” programming languages was a mistake in the sense that these baroque monstrosities, these conglomerations of idiosyncrasies, are really unmanageable, both mechanically and mentally. I see a great future for very systematic and very modest programming languages. When I say “modest”, I mean that, for instance, not only ALGOL 60’s “for clause”, but even FORTRAN’s “DO loop” may find themselves thrown out as being too baroque. I have run a a little programming experiment with really experienced volunteers, but something quite unintended and quite unexpected turned up. None of my volunteers found the obvious and most elegant solution. Upon closer analysis this turned out to have a common source: their notion of repetition was so tightly connected to the idea of an associated controlled variable to be stepped up, that they were mentally blocked from seeing the obvious. Their solutions were less efficient, needlessly hard to understand, and it took them a very long time to find them. It was a revealing, but also shocking experience for me.

                                                                1. 3

                                                                  I have been using i3 for the past half year or so on my work computer and have grown very fond of it.

                                                                  I might be something of a hoarder, I usually wind up with easily a dozen terminals, two editor windows, two browser, a chat, etc. I find that on typical drag-and-drop window managers I spend a lot of time (and mental energy) locating my stuff, shifting windows around, organizing stuff. I find that with a tiling window manager, I have almost no wasted space and an easy way to organize the windows such that what I am using often is close together and what is seldom used together is further separated. Instead of shifting windows the whole day, I typically set up my split once and keep it until my laptop crashes. If i3 hasn’t been a boon for productivity, it has at least been positive for not feeling burdened with mundane housekeeping tasks.

                                                                  That being said, i3 is not a desktop environment. You will find that when you start programs from the GTK or KDE families, things will look odd and will not be integrated, at times. I am not sure I could stomach this at home for a laptop that I also use for more leisurely activities. Also, nobody else can use your computer anymore. Whether this is a curse or a blessing depends ;)

                                                                  1. 3

                                                                    You will find that when you start programs from the GTK or KDE families, things will look odd and will not be integrated, at times.

                                                                    You can launch gnome-settings-daemon in your .xsession or whatever to handle your fonts and themes; that’s pretty lightweight and works regardless of what WM/desktop you use.

                                                                    1. 3

                                                                      I agree with this. As far as I’m concerned, common floating window managers are manual window manages. Splitting tiling window managers (like i3) are semi-automatic window managers. Layout-based tiling window managers (like xmonad) are automatic window managers. You simply cease managing windows completely, and leave that task to your… window manager.

                                                                    1. 9

                                                                      Sadly though, popularity is seldom a good indicator of quality.

                                                                      I mean, ought we really to learn about Soft Skills from a guy who demonstrates said skills like this, and who makes videos on YouTube about “getting the girls you want”?

                                                                      I’d be skeptical of anyone who describes themselves as a life coach, or who’s personal finance and investing advice is “just buy one house every year”.

                                                                      1. 2

                                                                        I think it’s not good as an absolute indicator of quality. I think it’s a useful signal and many of these books I’ve read, consider quality, and thus will consider other things associated with them.

                                                                        I like these lists for two reasons- they are easy to skim and find new ideas about things to read, they spur conversations where people provide additional info and recommendations.

                                                                        1. 1

                                                                          Since OP uses statistics here, my immediate thought upon reading the methodology is that more statistics ought to have been used to address these very questions.

                                                                          For instance:

                                                                          • If the recommendation rate for a particular book stays totally steady over time (especially if the same names are associated with it) then there’s probably a small but vocal fanbase doing the recommendations and failing to make an impact (which may mean that the book isn’t useful outside of a fringe audience, or may mean that people who are predisposed to like the book are also predisposed to being very bad at promoting it). I’d expect Knuth to be in this position: Art of Computer Programming is big, dense, and expensive, and only giant nerds will actually read it.
                                                                          • If the recommendation rate gets too high (especially if there’s a lot of variety in who is recommending it) and the problems it’s meant to address don’t go away, then it’s likely that the book presents seductive advice that doesn’t really work (or only works in trivial situations, or has extremely limited benefits). I’d expect the design pattern & testing books to sort of be in this category: both are extremely popular techniques, but their widespread adoption hasn’t really coincided with better-quality code because, while they make certain kinds of trivial mistakes more visible and therefore make certain kinds of debugging easier, they introduce their own complications (especially from naive application) and do not address the kinds of problems that experienced developers were already having the most trouble with.
                                                                          • I remember being assigned several of these books in college, & didn’t think much of them. The same way that your friend from high school who didn’t really read considers his favorite book to be The Great Gatsby (or some other piece of assigned reading that ended up being halfway decent), folks who don’t read CS books are liable to overestimate the quality of the few they’ve read, and ones that are assigned will be the only ones they’ve picked up. So, we can look at trends in recommendations compared to trends in assignment in syllabus, and see if they line up too closely. If the association is almost linear, then the books are probably not very good (and are merely marginally better than the average textbook).
                                                                          1. 1

                                                                            I agree. I didn’t know about Soft Skills and its author, but it’s pretty horrible.

                                                                            I am always quite surprised when I see The Art of Computer Programming in close proximity to the likes of Clean Coder.

                                                                            The first, to me, is about as close to a work of art that you can get in our field work. It’s a rigorous, didactic work, beautifully put together and playful at times.

                                                                            The second reeks of self help books void of content. Robert C. Martin, I refuse to call him “Uncle Bob”, campaigns for things I don’t believe are healthy for you, neither as a professional nor as a human being. A vivid example is his ideal to spend 40 hours of work per week “on your employers terms” and then 20 more “reading, practicing, learning, and otherwise enhacindg career”. The latter 20 hours are, in his words, “for you”. I think this is way too much and mostly in the interest of your employer.

                                                                            I wanted to write that I would not rate Robert C. Martin very close to John Sonmez of Soft Skills, until I found that Robert C. Martin wrote a foreword for the book.

                                                                          1. 7

                                                                            I’ve always loved this technique. There was a modification to the standard Lua interpreter that used this and it got something like a 20% speed up with no other changes.

                                                                            It does remind me of the old practice of using high/low bits of pointers for metainformation, and then as operating systems and hardware start using more and more of the pointers’ bits, you run into problems.

                                                                            For example, early MacOS used the high bits of pointers for various flags. This worked because the original hardware (M68k) had 32-bit pointers, but only 24 address lines, so masking off the top bits didn’t matter. As later processors in the family used more bits, this suddenly became and issue and software had to be rewritten.

                                                                            The technique discussed here assumes that 64-bit pointers will fit into 48 bits. This is currently true, but all of the major 64-bit architectures reserve the right to extend how many physical bits will end up being used regardless of how many are today; IIRC, IBM z doesn’t make any distinction and states that all 64 bits may be used today.

                                                                            1. 3

                                                                              When I was first getting into Lua circa 2005, I found out about how it represents values and felt like there had to be some way to scrunch them smaller than 16 bytes. I was envisioning something like NaN tagging, but I didn’t know enough about floating point. It was run to run into it the other day and know that people had gotten it working!

                                                                              I think tagged pointers originated with early LISP implementations. The various LISP machines that were built over the years sometimes had custom memory architectures with extra bits reserved for tags!

                                                                              1. 2

                                                                                Wait might be dense here but.. so if a program requires/requests a clean 8 bytes of stack/heap mem, and gets it to use it however it pleases, surely the actual meaning of those 8 bytes (address-value or other content) cannot possibly be messed with by the OS and/or the arch. Are you talking only about pointer values retrieved from OS/kernel calls? Yeah well maybe those shouldn’t be stuffed with custom cookie crumbs. But own pointers to own locations? Who or what, outside the program’s code, will mess with any of those bits?!

                                                                                1. 4

                                                                                  You’re right: if you’re using a custom allocator, you’ll be fine for most cases if you can control the upper bits of the allocation range. For example, if you use a custom allocator based on an mmap‘d heap and request that the address that the new heap gets mapped to is guaranteed to have all its top n bits be zero (which necessarily limits the size of your heap), then you can use the trick from TFA treat those bits however you want and life’s good, so long as your heap stays addressable in 48 bits.

                                                                                  But if you interface with other libraries or things allocated by the system, this is not technically future-proof. Of course, it’s not going to be a problem for a long time if ever for most platforms (but not all…). It’s just reminiscent of the old MacOS issue.

                                                                                  (Note that this technique technically won’t work even now on certain platforms: I don’t think it would work consistently on IBM i or z, though that’s just a guess, and SAOS’s on SPARC or POWER might also do something funky….but for mainstream systems on popular hardware this works fine.)

                                                                                  1. 1

                                                                                    Ah yeah of course all heap addrs come from the system in the final analysis one way or the other. Yeah I can see the issue more clearly now

                                                                                  2. 3

                                                                                    If you align everything to 8 bytes, as you say, you get 8 combinations or 3 bits to fill with whatever information you please.

                                                                                    As you also pointed out, when you interface with something that doesn’t know about your tagging convention, you have to strip away the tagging bits before passing it to that something.

                                                                                    Apart from that I don’t see how the OS or your architecture might mess with your pointers.

                                                                                    What might be an issue, however, is fighting the compiler. While sound from a processor/architecture perspective, the tagging is undefined behavior according to C/C++ standards. You have to compile with flags to get the desired behavior with certainty.

                                                                                    1. 2

                                                                                      The problem comes up when you try to store additional data types without extra storage: assume that all pointers will have the top x bits unused and set to 0. This means that you have 2^x other namespaces of the same size without taking extra space.

                                                                                      For example some smalltalk VMs use a bit to identify if a word contains a pointer or a number, reducing both the address space and the native number range to 1 bit less than what the architecture supports natively. That works as long as you can control the address space (these VMs tend to work on a big pre-allocated slab of memory). In some scenarios like m68k macos the limit was lifted underneath the program (thanks to newer processors) and so the trick failed spectacularly because there were now pointers that looked to the program like something else.

                                                                                  1. 10

                                                                                    This article very accurately describes how I approach understanding systems. I would like to write a long comment, but I feel like almost everything I wanted to say has already been said better by the author.

                                                                                    One specific point I had on my to-write-about list for a long time now is this:

                                                                                    Learning more about software systems is a compounding skill. The more systems you’ve seen, the more patterns you have available to match future systems against, and the more skills and tricks and techniques you develop to apply to future problems.

                                                                                    I mentally describe this as a logarithmic learning technique. When you have a system with n layers on the path you are looking at, a reasonable guess about the size of the state space you’re dealing with is to say that it is “on the order of exp(n)”. Clearly, thinking about such a system as whole does not work for any non trivial value of n. However learning and thinking about a core set of rules and behaviors of the involved layers only takes work “on the order of n”, which is the same as log(exp(n)), hence the name.

                                                                                    As an example, I once debugged a bug where a file transfer through a proprietary RPC failed for certain files at the very end of the transfer. Through the use of Wireshark a reasonable guess was that TCP segments - and hence Ethernet frames - of a certain length would not get ACKed. Doing a file transfer involved a number of layers I had little knowledge about.

                                                                                    However by looking at the layering of the RPC protocol it was not a monumental task to deduce how files get split into packets, framed with some meta information and then written to a TCP socket before waiting for an answer from the remote side. This allowed a prediction which files, meaning of which sizes, will not get properly transferred. We were able to pad files of offending sizes and thus provide a hot fix. In a second step this allowed a very simple repro case to be written: ping with a -s size to get ethernet packets of 1498 or 1499 bytes in size. From there it was easy to convince our vendor that yes, the ethernet driver was broken and they’d fix it.

                                                                                    1. 3

                                                                                      Unfortunately I don’t have a good answer on how to approach this, but I am feeling your pain.

                                                                                      I have the impression that this kind of documentation is often written by developers who are deeply immersed in the concepts. This means that they will primarily speak about how their specific software works, not about the concepts. In your example, they at least have this section “3.1 Basic Linker Script Concepts”, but it’s pretty limited. I’m not convinced I would have the right grasp on the concept of “load address” after I read only this.

                                                                                      As I said, I haven’t figured out good and quick way to deal with the kind of problem you are describing, i.e. missing overview and hard to look up names, concepts or functions. What works for me, but obviously takes a lot of time, is to do a bit of everything in the list at the same time:

                                                                                      • Read the documentation of a linker or multiple linkers. That’s what you started out with. However, don’t read everything but also do the things below and periodically come back to the documentation.
                                                                                      • Read tutorials, you might learn names of concepts from them.
                                                                                      • Come up with projects you know that have similar requirements to linking as your. Then check their linker scripts how they solved the problem. Here you can learn common tricks and idioms.
                                                                                      • Play around with small examples, try to build a very small program for your platform and move different parts of the program to different places, just to see whether you can make it. This will provide you with a feeling on how the parts fit together.
                                                                                      1. 18

                                                                                        I don’t think we should change the protocols and force every library in every language on every platform to update mountains of code to support a new protocol just so my browser can download Javascript trackers and crappy Javascript frameworks faster.

                                                                                        1. 17

                                                                                          I’m excited for HTTP/3 because it will allow me to get lower-latency video streaming support for my private stream server.

                                                                                          1. 15

                                                                                            Well, just like with HTTP/1 and /2, the old protocols are very likely to be supported for a very long while. So you’re not forced to update.

                                                                                            1. 12

                                                                                              It’s still change just for the sake of allowing people to build even more bloated websites.

                                                                                              Making HTTP more efficient isn’t going to mean websites load faster, it means people are going to stuff even more tracking and malware and bloat into the same space. It’s very, very much like building bigger wider roads with more lanes: it doesn’t alleviate congestion, it just encourages more traffic.

                                                                                              1. 27

                                                                                                I don’t think that’s entirely true, HTTP/3 does address some problems that we have with TCP and HTTP in modern network connections. I encounter those problems every day at work, it’s just background noise but it annoys users and sysadmins.

                                                                                                1. 14

                                                                                                  As I understand that video, HTTP/3 is not a new protocol, but rather “HTTP/2 over QUIC”, where QUIC is a replacement for TCP. QUIC can be useful for a lot of other applications, too.

                                                                                                  People do a lot of stuff to work around limitations, like “bundling” files, image sprites, serving assets from different domains, etc, and browsers work around with parallel requests etc. So it saves work, too.

                                                                                                  Whether you like it or not, there are many WebApps like Slack, GitHub, Email clients, etc. etc. that will benefit from this. Chucking all of that in the “tracking and malware”-bin is horribly simplistic at best.

                                                                                                  Even a simple site like Lobsters or a basic news site will benefit; most websites contain at least a few resources (CSS, some JS, maybe some images) and just setting up one connection instead of a whole bunch seems like a better solution.

                                                                                                  1. 8

                                                                                                    Don’t you think that people are going to stuff even more bloat anyway, even if everybody downgrades to HTTP/1?

                                                                                                    1. 6

                                                                                                      I don’t know that people will drive less if you make the roads smaller. But they won’t drive as much if you don’t make the roads bigger in the first place. They’ll drive less if you provide bike lanes, though.

                                                                                                      In an ideal world AMP would be like bike lanes: special lanes for very efficient websites that don’t drag a whole lot of harmful crap around with them. Instead they’re more like special proprietary lanes on special proprietary roads for special proprietary electric scooters all vertically integrated by one company.

                                                                                                2. 9

                                                                                                  The old protocols over TCP provide terrible experiences on poor networks. Almost unusable for anything dynamic/interactive.

                                                                                                  1. 1

                                                                                                    TCP is specifically designed and optimised for poor networks. The worst networks today are orders of magnitude better than the networks that were around when TCP was designed.

                                                                                                    1. 13

                                                                                                      There are certainly types of poor networks that are ubiquitous today that TCP was not designed for.

                                                                                                      For instance, Wifi networks drop packets due to environmental factors not linked to congestion. TCP data rate control is built on the assumption that packets are dropped when the network is congested. As a result, considerable available bandwidth goes unused. This can qualify as a terrible experience, especially from a latency point of view.

                                                                                                      If your IP address changes often, say in a mobile network, you lose your connection all the time. Seeing that connection == session for many applications, this is terrible.

                                                                                                      Also many applications build their own multiplexing on top of TCP, which, constrained by head of line blocking, leads to buffer bloat and a slow, terrible experience.

                                                                                                      1. 5

                                                                                                        Related to this:

                                                                                                        https://eng.uber.com/employing-quic-protocol/

                                                                                                        Mobile networks are a prime target for optimizing latency and minimizing round trips.

                                                                                                      2. 1

                                                                                                        It was designed when latency didn’t matter. Now it does matter. Three-way handshakes and ACKs are killing us.

                                                                                                        1. 1

                                                                                                          It seems to me that every reasonable website I use is fine with those tiny inefficiencies because they’re generally efficient anyway, while bloated malware-filled tracking javascript-bloated nightmare websites are going to be bad either way.

                                                                                                          Who is this actually helping?

                                                                                                          1. -2

                                                                                                            It’s helping people with actual experience in this area. Please stop posting these hyperboles and refrain from further comments on this topic. You’re wasting everyone’s time with your hot takes.

                                                                                                            1. 0

                                                                                                              Leave the moderation to the moderators. My opinions are pretty widely held and agreed with on this issue. Degrading them as ‘hot takes’ is unkind low effort trolling.

                                                                                                              If you have a genuinely constructive comment to make I suggest you make it. If you don’t I suggest you stay quiet.

                                                                                                              1. 1

                                                                                                                I do not refute that there are issues with tracking and malware but if you think we are going to regress an era without a rich web you are out of your gourd. There is no future where the web uses fewer requests. The number of images and supporting files like JavaScript will only increase. JavaScript may be replaced in the future with something equally capable, but that still will not change the outcome in any appreciable way.

                                                                                                    2. 6

                                                                                                      Without even talking about HTTP/3, it seems that any application that uses a TCP or UDP connection could benefit from using QUIC: web applications yes, but also video games, streaming, P2P, etc…

                                                                                                      Daniel Stenberg also mentioned that QUIC would improve client with a bad internet connection because a packet loss on a stream does not affect the others, making the overall connection more resilient.

                                                                                                      I do agree it could and will be used to serve even more bloated websites, but it is not the only purpose of these RFC.

                                                                                                    1. 1

                                                                                                      A little tongue in cheek, people approach this also from the other side as well, i.e. https://stackoverflow.com/questions/415434/can-automated-unit-testing-replace-static-type-checking

                                                                                                      1. 1

                                                                                                        Nice find, and it makes sense to me that there is overlap between testing and typing. Things like dependent types or Clojure spec blur the lines even farther.

                                                                                                      1. 13

                                                                                                        I2C sounds simple and powerful, but I have grown quite resentful of it. This stems from a decade of having to work with it - debugging is the worst.

                                                                                                        There are a myriad of issues with the bus, the worst being that it can get stuck - and there’s no way to get it out of that state. A comment below the article has already mentioned this. Of course you can wire resets to all the bus devices, but that increases pin count from 2 to 3, not counting GND, and is outside of what the I2C bus standard entails. Of course you can try to wiggle the bus a bit and see if this frees it from a stuck state. Maybe now you have an unwanted write to an EEPROM. Or maybe the method doesn’t work because a stuck slave is clock stretching indefinitely. The list can be made to go on for quite a while.

                                                                                                        My personal advice for those considering to use I2C: Don’t, unless you are forced to. If you still think you want to use, the following checklist will help you have a better experience.

                                                                                                        • Have only a single master. I have never seen a trustworthy multi-master system.
                                                                                                        • Have only “simple” slaves. I.e. EEPROMs with hardware implemented I2C interfaces without clock stretching are OK. Microcontrollers are not OK, the program can get stuck and you wind up with that slave stretching the clock indefinitely. Be aware that if you buy, say, “an IMU”, it may actually contain a microcontroller handling the I2C communication. This is the case for a number of more complex devices.
                                                                                                        • No hotplug.
                                                                                                        • Have all devices powered from the same supply.
                                                                                                        • Have few devices on the bus.
                                                                                                        • Have a short bus.
                                                                                                        • Nice to have: Be able to reset slaves from the master.
                                                                                                        1. 2

                                                                                                          Great comment, and your checklist is dead on.

                                                                                                          One thing I will say though: I’ve shipped ~10 CE products, all of them with I2C devices, and have only had problems with one peripheral: Apple’s MFi chip (used to be required for BT accessories).

                                                                                                          So while in practice I2C buses can get stuck, it is not a problem of epidemic proportion as some would have you believe.

                                                                                                          Still - it’s good practice to have a plan to recover in the event the issue crops up. Reset pins & power control are the way to go.

                                                                                                          Last but not least, always make sure the electrical characteristics of your bus are correct. If transition or hold times are violated, you’re going to have a bad time.

                                                                                                          1. 1

                                                                                                            One thing I will say though: I’ve shipped ~10 CE products, all of them with I2C devices, and have only had problems with one peripheral: Apple’s MFi chip (used to be required for BT accessories).

                                                                                                            Out of sheer interest: How well did these products match the checklist?

                                                                                                            How big a problem you have with I2C depends a lot on your design.

                                                                                                            If you have, say, an I2C bus with an imaging sensor, a thermometer and an EEPROM, you might be able do dodge the worst of troubles. If “the bus gets stuck” and you have appropriate measures in place to reset all devices, your device might carry on with only a small hiccup. To do this requires forethought in the hardware design and you have to invest into your software. On the hardware side, you need resets to the chips that support resets, your imaging sensor most certainly. The EEPROM (like 24xx devices) has no reset pin. If the bus interface is not responding, you have to power cycle it. This takes up board space and you have to get the power supply right. On the software side you need to detect that “the bus is stuck” and you need to act. Then your software needs to deal with hardware not always being reachable, maybe even being partially reachable.

                                                                                                            I think that it would be much more sensible to have a communication standard where, if communication with one component fails, only communication with that one component fails.

                                                                                                            One project I worked on had a central processor and several microcontrollers scattered throughout the system, all connected through I2C. One of the microcontrollers was the first to start and controlled power sequencing for startup, standby and shutdown. It also had an IMU connected, the data of which the main processor needs to read. When reading IMU data, bus traffic increases considerably, triggering a (as I believe, still unsolved) preexisting within a minute instead of within hours. The fault was always shortly after some microcontrollers were powered up and hotplugged to the bus. The bug leads to the power sequencing microcontroller not serving I2C anymore, infinitely clock stretching. The main processor canniot reset the power sequencing uC for different reasons: it does not have a connection to it, apart from the stuck I2C and the power sequencing uC has to be the last system part to be powered down.

                                                                                                            Now, clearly this hardware design is bonkers, for a variety of reasons. The power sequencing uC should have fewer responsibilities, i.e. only power sequencing. IMU readout should be ideally with another uC and clearly on a bus where it’s isolated from more critical functions. Also very clearly, the design is bonkers because it violates several points on the I2C checklist and insists on using I2C for vital system parts.

                                                                                                            Last but not least, always make sure the electrical characteristics of your bus are correct. If transition or hold times are violated, you’re going to have a bad time.

                                                                                                            This is true. However, in many cases you don’t have control over all the conditions that you need to control in order to guarantee reliable operation.

                                                                                                            The I2C standard is not really a standard, it’s more like a leisurely application note. This leads to many different interpretations and assumptions about how the bus is driven, assumptions that may not be true. As a result, connecting different, unknown devices together leads to a significant amount of new and exciting bugs.

                                                                                                            To sum this us, my advice remains the same: Don’t use I2C unless you have to. If you have to, fulfill the checklist as good as you are allowed to.