1. 32
  1. 9

    I submit as a counterpoint that, in yosefk’s experience, low-level is easy.

    1. 4

      Yeah I like this. One observation I’ve had: writing C or C++ is relatively easy, but the corresponding Makefiles and autotools are harder.

      Does anyone else feel that way? Once you are situated and understand pointers and all that, it feels pretty easy to write the actual C or C++ code. Particularly if you are only using libc and libstdc++ and not huge abstractions like APR or OpenGL or something (low-level is easy).

      But the hard parts seem like getting the compilers and build system to do what you want… debugging preprocessor directives can be harder than debugging the program. Getting the debugger to run against a binary with the right build flags (for an arbitrary open source project) is harder than debugging it.

      For one, a Unix-style build system for a mature project requires understanding 3 or 4 different languages: shell, make, sometimes awk, {m4/autotools, cmake, Ninja} etc. Whereas C and C++ is 1 or 2 languages.

      1. 3

        Learning how to use all tools required to be a truly effective C programmer may indeed be approximately as hard as learning basic C.

        1. 3

          Where I work, embedded code that runs on our cameras is generally easy to understand, and reads like a captivating yet approachable exploration in low-level systems programming. Our marginally technical product managers can understand this code, and some indeed do send in PRs against it.

          On the other hand, our externally facing APIs and various data pipelines are a morass of abstractions and concessions to past scalability problems that make even the gnarliest embedded code look like child’s play. There’s a joke around the office that goes something like “embedded tends to suck in everyone from time to time”, but I suspect that people go there to be productive in a simpler, more straightforward world.

          1. 2

            One observation I’ve had: writing C or C++ is relatively easy, but the corresponding Makefiles and autotools are harder.

            Particularly if you are only using libc and libstdc++ and not huge abstractions like APR or OpenGL or something (low-level is easy).

            Those two statements are almost contradictory.

            If your project only depends on libc and libstdc++ and its not using any third party libraries then it’s nobody’s fault but your own if your Makefile is very complicated. Using autotools in that situation buys you nothing but extra complexity and a fancy Makefile.

            IME it’s much easier to create a Makefile template with the targets you want, and then reuse it for small projects. The Makefile syntax itself can be complicated, but 99.99% of the time it’s best to keep it simple.

            1. 2

              Point taken, but if the thesis is “low level is easy” then you can make the comparison between plain C code with no deps, and a huge generated makefile for something with deps.

              In both cases, the higher level layers are what make things hard. I concede that using and understanding the big platform layers in C are certainly more difficult than a makefile.

              I guess the overall point is that on a typical big C project you have two sources of complexity and dofficulty – build time and runtime. Runtime is to some extent essential, but the mess at build time can be fixed.

              1. 2

                Even with simple code and no dependencies, you need a huge messy Makefile if you want to support debug/asan/release builds on win32/win64/Linux/OSX. (doing make clean and changing CFLAGS whenever you want to switch builds is not good enough)

              2. 2

                writing C or C++ is relatively easy

                I don’t feel this way. Writing C that never trips undefined behaviour does not feel particularly easy to me. Also I have to churn out a lot of C code to get it to do fairly trivial things, and then I have to find all the stupid mistakes I left in.

                I don’t think the actual writing of low-level software is itself easier. You’re implementing algorithms on your hands and knees all the time. You so much as look at cc funny and you get undefined behaviour you might not discover for years. The debugging tools are often broken.

                I think yosefk is arguing that being a software developer working on low-level software is overall easier, because the part where you actually write the software is somewhat harder, but everything else involved in the job is easier. You have far less difficulties caused by, for example, people handing you ill-defined requirements to implement.

                I think the important bit of the article I linked is this:

                As a low-level programmer, you have to convince people not to be afraid when you give them something. As a high-level programmer, you have to convince them that you can’t just give them more and more and MORE.

                1. 2

                  Well, that’s on you for using autotools and their ilk :)

                  Projects under 50,000 lines or so only need a build script like this:

                  cd build && clang -Wall -flto ../*.c ../*.cpp

                  For larger projects, modern build systems that output ninja are relatively painless.

                  1. 3

                    It’s relatively painless, but not even close to actually painless.

                    You shouldn’t need a several hundred line script to output a several hundred line build config that a ten thousand line build tool parses to run your compiler. But, C doesn’t include any way to specify how your code should be built and our compilers are crap so incremental and parallel compilation are must haves.

                    Unity builds solve the problem of needing massive build tools, but your code has to be written to work as a unity build from the start and using 3rd party libraries becomes a big pain. It also doesn’t really solve the performance issues but it does help a bit.

                    Compiling your code is one of the biggest pain points with C, and the Jai language is aiming to resolve this by making the build specification part of the language and having a compiler that can build hundreds of thousands of lines per second.

                2. 2

                  Do you think systems programming is low-level? This is an honest question – whenever I read something about OS or systems development, like distributed platform development, I see fairly high-level things. Hardware development is low-level though… but what are systems if not hardware abstractions, so by definition, a higher level thing?

                  1. 3


                    Specifically about this James Mickens' essay here: the essay is about the implementations of kernels, virtual machines, device drivers, databases and the like. The pain this essay discusses is specifically the pain of low-level programming, such as how bugs in the thing you’re implementing can also break the tools that you’d like to use to debug it, such as a stray pointer breaking your logging subsystem and now, in Mickens' words, “I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.”

                    By definition: I think systems programming is “writing stuff intended for applications to be built on top of” so it’s more low-level than applications programming, which consumes the software that systems programmers produce.

                    1. 3

                      An example I have seen is someone fixing a localization bug, and added logging to debug/verify that it’s working. However, logging actually called into his localization code which infinitely indirectly recursed, causing the system to hang on boot. This ain’t quite an example of low level debugging but it was low enough to break tools your tools rely on.

                    2. 3

                      In yosefk’s sense, low-dependency code is easy. This describes quite a lot of, but not nearly all, systems code. (E.g. a special case in the PostgreSQL query planner probably isn’t low-dependency.)

                  2. 5

                    Title should just be “The Night Watch”, and probably have the satire tag.

                    1. 4

                      I don’t think ‘satire’ is a good description of this.

                      1. 3

                        It just happens to be the closest tag which lobsters currently has to fit “funny writing”. There’s also a “rants” tag but Mickens' style is IMO much more audience amusement than catharsis.