Threads for borisk

  1. 9

    Great read. I started off wondering why anyone would need to replace something as “obviously simple” as pkg-config, and finished the article completely horrified by pkg-config‘s behaviour. And it’s not even bootstrappable.

    1. 13

      I came from a different starting point: I was watching closely when FreeBSD switched from pkg-config to pkgconf and so I’m fully aware of how horrible the original codebase was (to work on and to build), but I didn’t know why the author didn’t just use pkgconf.

      I wasn’t tremendously impressed with the code quality in pkgconf either. It uses fixed-width bitfields in a few places with no abstraction and with all of the bits allocated so if you need to add a new flag later it’s a massive refactoring, for example, so I can kind-of see why you’d want to write a new version.

      The thing that I really didn’t understand is why you’d use C, particularly if Windows support is a key requirement. Unlike the C standard library, the C++ standard library is very well supported on Windows (by both Visual Studio and LLVM’s libc++) and provides platform abstractions over all of the things that are required for a pkg-config implementation. If you’re going to write a tool that does a load of string processing, why not implement it in a language that at least has a standard-library string type that integrates with its I/O functionality, even if it’s not a great one). Personally, I’d prefer to see pkg-config written in Lua with a tiny C++ wrapper for launching it. It’s not like it’s performance critical.

      1. 6

        As poor as the original pkg-config’s code and bootstrapability may be, I haven’t hit any bugs with it yet while with pkgconf I’ve been lucky enough to find a “how is the code structured that makes this possible” bug: https://github.com/pkgconf/pkgconf/issues/275

        Totally agree that lua or similar would be more appropriate for this kind of program.

        1. 4

          I wasn’t tremendously impressed with the code quality in pkgconf either.

          It’s an over-engineered, inscruitable mess and its author has no regard for backwards compatibility. It got so bad that we forked the library, cleaned it up a bit, and fixed quite a few buffer overflows. In case anyone is interested: https://github.com/build2/libpkg-config It builds and works fine on Windows.

          BTW, the README file in the above-linked project also describes the alternative usage from a build system that addresses quite a few issues (like cross compilation) that plague the traditional pkg-config usage.

      1. 4

        I’ve shared this mainly because I find interesting that the project is moving from C++ to C.

        1. 2

          Cross-platform multithreading in C will be fun. While the author doesn’t plan to start own threads, they do plan to use less-than-pointer handles, which means some sort of a lookup table and thus synchronization.

          1. 2

            they do plan to use less-than-pointer handles, which means some sort of a lookup table and thus synchronization

            Theoretically that doesn’t have to be the case (but probably is). It’s possible that the library will use one block of memory (or one per type) and use those handles to index into that. In such case no synchronization primitives would be needed when accessing such objects since the conversion to real pointer is just base pointer and handle addition. Considering that the author talks about pinning I doubt that they will go with such approach.

        1. 3

          My big problem with SemVer is that it’s a property of interfaces not implementations and so using SemVer for package versions is inappropriate. I’d be very happy with a package using calendar versions for its releases and advertising the set of interfaces that is supports with semantic versions.

          COM handles interface versioning in the simplest way possible: any change is considered a breaking change and needs a new version. Languages that support adding APIs (new types in a namespace, new methods in a class, and so on) in a non-breaking way[1] can support additions in a minor revision and removals only in a major revision.

          Good software engineering with graceful deprecation periods is impossible with the semver-for-implementations model. Imagine three versions:

          • A includes a feature.
          • B deprecates the feature and adds a replacement.
          • C removes the deprecated feature.

          This is the flow that I want from libraries that I consume. I move from A to B, test, get some warnings, eventually migrate to the new API, then migrate from B to C. At any given point, my code either works with A and B, or with B and C. With semantic versioning for interfaces this works fine:

          • A implements 1.0
          • B implements 1.1 (or 1.0 if the replacement feature is not exposed on the old interface) and 2.0
          • C implements 2.0 (or 2.1 if it also adds some new things)

          With implementation SemVer, you end up with C being 2.0, indicating that it’s the one that’s a breaking change, but it isn’t a breaking change for anything that heeded the deprecation warnings and moved to the new APIs when using B.

          [1] This is tricky in the presence of patten matching on types and other forms of introspection, because some code may depend on the absence of a method with a particular name on a type.

          1. 1

            Can’t this be approximated by creating a differently-named package for 2.0? As in, you start with libfoo 1.0, then release libfoo 1.1 with the deprecation warnings along with libfoo2 2.0 which implements the new API. I suppose there won’t be a version that implements both APIs at the same time but maybe that’s a good thing (KISS and all)?

            1. 1

              That would work as long as you have a single user of the package. If not then you will end up linking libfoo and libfoo2 in the same program, which will often cause problems. With versioned interfaces, any library that depends on libfoo can move to the 2.0 API and expose compat interfaces that the the 1.0 interfaces from libfoo, allowing you to upgrade each library independently and then move to the C version once everything has moved. With the libfoo and libfoo2 distinction you end up with a flag day when you must bump all dependencies to a new version at the same time. In my experience, this kind of flag day is one of the worst things to hit in software engineering because something will invariably break and it takes ages to find the root cause because you’ve changed too many things at once.

          1. 9

            I don’t dislike make, and even use it quite a lot, both at work and outside, but let’s be realistic here:

            It’s already available everywhere.

            No, it’s not, not on Windows, where even if you have something like Git Bash, it won’t have make. And even after installing it, using it on windows is extremely annoying, specially if you want to use PowerShell.

            Anyone telling me to “well, don’t use Windows” will be cursed with 10 years of intermittent production bugs that don’t leave stack traces.

            It’s fast.

            That much is true.

            It’s language-agnostic

            Ish. It’s not shell agnostic, at least not enough to support something like PowerShell, which means your target rules still need to be in shell script. Which sucks as soon as anything more complex is needed.

            It’s simple (…)

            LoL. Maybe there’s a BSD/POSIX make that is simple, but, again, realistic, everyone uses GNU make, and that is not simple. It certainly doesn’t make it simple to do most things you might want to to with it if you’re not compiling C, which is my case, and the article’s case.

            Make is one of those things that when you stop to really look at it, is not actually good for most of the things we hammer it into. It’s just less bad and less inconvenient than all other alternatives.

            1. 7

              To be fair, he says make is a bad choice if you need to use Windows.

              1. 3

                That’s what I get for skimming instead of reading with attention =P

              2. 4

                No, it’s not, not on Windows […] It’s not shell agnostic, at least not enough to support something like PowerShell, which means your target rules still need to be in shell script. Which sucks as soon as anything more complex is needed.

                Very true. This was one of the main reasons for us starting build2 – we needed Windows to be the first class citizen. To achieve this we had to invent our own shell-like language (called Buildscript; while at it we’ve also fixed a couple of major POSIX shell annoyances like having to quote variable expansions or not failing on errors by default). Here is an example of a rule written in Buildscript. It works the same everywhere, including on Windows.

                Having said that:

                Anyone telling me to “well, don’t use Windows” will be cursed with 10 years of intermittent production bugs that don’t leave stack traces.

                You mean the same as if you do use Windows? Supporting Windows in build2 was and is a constant source of pain and frustration. So if you can avoid using Windows, by all means do. You will save yourself a lot of grief.

                1. 2

                  I’ll check that out, sounds interesting!

                  You mean the same as if you do use Windows

                  Hehe, I feel you. But yeah, I don’t love windows, but I’m currently working in a bank, I don’t get much leeway I’m choosing technologies, hence the comment XD

                2. 2

                  So, at several companies now I’ve worked with Makefile-driven setups, and even been one of the people responsible for building and maintaining them.

                  As a developer experience, it is of course not perfect, but it’s also better than a lot of the alternatives. If what you want is a tool that lets you specify a set of tasks to execute and have all the developers on the team be able to use it, well, basically everything that can be leveled as criticism at make can also be leveled at other automation/task orchestration tools. Lots of them are not fully cross-platform, or require extra installation/upkeep, or have limitations due to being designed with a certain specific task domain in mind, etc. etc. etc,

                  As a lowest-common-denominator of “available” and “relatively easy to write the config for”, make does a pretty decent job.

                  Anyone telling me to “well, don’t use Windows” will be cursed with 10 years of intermittent production bugs that don’t leave stack traces.

                  Remember that we’re talking about web development here, so there aren’t Windows-specific builds – in my own case it’s all backend services that will deploy to containers. And the entire team is on various flavors of Unix-ish operating systems, with no developers on Windows as their daily driver OS. So this is a purely irrelevant objection.

                  1. 1

                    So this is a purely irrelevant objection.

                    Bit harsh. Bit of a fallacy, too: you or your team not using windows doesn’t mean no one does it, anywhere.

                    1. 1

                      Neither harsh nor fallacious. Your whole objection seems to be asking what to do about developers who are on Windows and builds that are broken on Windows. But the whole point of my comment is: there are teams out there who do not have “Windows builds” and who have no developers on Windows.

                      If I were building software that require OS-specific builds, or working with people who were on Windows, I’d choose a different task automator. But I’m not, so make poses no Windows-related problems to me.

                      And the entire context of the main article is web applications, which are platform-independent, so the objection of a build breaking/failing on Windows is… irrelevant, because a web application does not have platform-specific builds like that.

                      1. 1

                        I’m not sure why you started talking about broken builds? I didn’t mention it, and the makefile in the article has more PHONY targets than file dependent ones, so it’s using make as more of a task runner than a builder. Which is fine, I do that too.

                        Your whole objection seems to be (…) builds that are broken on Windows.

                        It isn’t, though? My objections were mostly that make isn’t really available everywhere, nor is it really simple.

                        web applications, which are platform-independent

                        The application can be as platform independent as you want, but unless you’re also doing all your development in a browser, your development environment isn’t. Your tools have to work with every platform that anyone in your team uses. And there are a lot of teams using Windows out there.

                        By the way, I did see, after writing the first comment, granted, that windows is mentioned at the end of the article, so it seems like the article itself puts at least an asterisk on the whole “make is available everywhere” argument.

                        1. 0

                          I’m not sure why you started talking about broken builds? I didn’t mention it,

                          You started off with:

                          Anyone telling me to “well, don’t use Windows” will be cursed with 10 years of intermittent production bugs that don’t leave stack traces.

                          That seems to be a pretty clear claim about builds that are broken on Windows. Which, for the third(?) time now, isn’t a thing with web applications because they don’t have platform-specific builds.

                          Your tools have to work with every platform that anyone in your team uses. And there are a lot of teams using Windows out there.

                          And there are equally a lot of teams not using Windows out there. Like the teams I work with. Like (presumably) the teams the author of the article works with. Dev tooling being non-portable to Windows is a non-issue if nobody on the dev team is using Windows. If a team doesn’t use Windows, it is OK for them to use some tools that are not portable to Windows. Really. I’m not sure why this seems to be such a difficult point to get across to you.

                          1. 1

                            You started off with (…)

                            That was a joke, completely independent from the main point. It’s not “clearly” about broken builds on windows at all, it’s about snarky responses that can come up when someone mentions using windows.

                            And there are equally a lot of teams not using Windows out there.

                            We’re talking in circles here. I’m not saying there’s someone using windows in every single development team in the known universe, I’m saying “some people use Windows”.

                            I don’t know why you think answering “some people use Windows” with “some people don’t!” is some kinda of argument that proves that I’m completely wrong? It’s not even a disagreement, it’s a logical conclusion: if only “some” people use Windows, necessarily there are people that don’t.

                            (…) is a non-issue if nobody on the dev team is using Windows.

                            You keep repeating this and I am out of ways to try to communicate that I am not talking specifically about your team, or any other you worked with in the past, I am talking in the general sense.

                            Can we agree on disagreeing? Because at the end of the day I’m pretty sure we’re both just gonna keep using make anyways, so, it’s a pointless discussion.

                            1. 1

                              You argued pretty strongly against Makefiles on grounds of non-portability to Windows. All I’m pointing out is that in the particular field of programming the original article’s author was discussing, this is often a non-concern – in web development, you don’t need a Windows-specific build of your application, and the number of developers who use Windows as their daily operating system for doing dev work is extremely small.

                              So basically you weren’t arguing against the article; you were arguing against a hypothetical alternate universe version where the article’s author worked in a field of programming where Windows support is much more important. And I said that’s not really relevant, because it isn’t relevant.

                1. 13

                  Mistake 1: Not using sdl2-config

                  Mistake 2: Including SDL2/SDL.h

                  The overarching mistake here is not using a proper build toolchain (build system and package manager). I especially take issue with this assertion:

                  The correct SDL2 include is the following:

                  #include "SDL.h"
                  

                  Including headers with the library prefix (#include <SDL2/SDL.h>) is our only hope to have an inter-operable ecosystem of C/C++ libraries. Let me illustrate this with a concrete example. What the article suggests is having the header search path (i.e., -I option from sdl2-config) pointing to the location of the SDL.h header (say, /usr/include/SDL2). But there are other headers in there. Thankfully, most of them have the SDL_ prefix (by all means an exception rather than the rule, when it comes to C/C++ libraries), but not all. For example, there is the generically-named begin_code.h header.

                  Imagine now that your project is using another library with the same arrangement which also contains a bunch of generically-named headers and one of them happened to be called begin_code.h. Now which one gets picked up depends on the order of the -I options that are passed to the compiler.

                  1. 2

                    Yeah, that advise seemed puzzling to me. Even sdl2-config - in a world with CMake and pkg-config…

                  1. 1

                    To me this feels like a perfect fit for FoundationDB (which has an in-memory storage backend so both “fits”/“doesn’t fit” scenarios could be benchmarked). Would have loved to read about an alternative implementation based on FoundationDB, if the author would be interested in trying one (I believe there is a FoundationDB client for Rust).

                    1. 1

                      While the article lists other valid reasons you might want to do this, there is no guarantee this will speed up compilation since doing this reduces parallelism. And this will be more and more so as we get more and more but slower cores in our CPUs.

                      1. 1

                        Finally, for ultimate peace of mind, use a language that has a less insane build toolchain than C++.

                        Or use a sane C/C++ build toolchain. For example, build2 handles pretty much every point mentioned in the article which is concerned with building, including static library dependencies. The overall complexity is still there, mind you, and it goes like 2x once you throw Windows into the mix, but that’s the world we live in for better or for worse (for some reason all the operating systems widely in use today heavily rely on shared libraries).

                        1. 4

                          I find the fact that things are performing worse after disabling bounds checks unsatisfying. IME, it’s not uncommon to get worse numbers after an optimization for a few runs but you should be able to at least get the same numbers as before after a few more runs, provide the optimized version does strictly less work compared to the original (as is presumably the case here). I think something else is going on here that we don’t understand.

                          EDIT: here is one theory that tries to explain the slowdown.

                          1. 28

                            This is an important lesson. Rust is aimed at being a safer systems language. There are a lot of safe applications languages with more mature ecosystems, better tooling, and so on. If you are in a situation where you don’t need fine-grained control over memory layout, where you can tolerate the memory and jitter of a global garbage collector, then Rust is as much the wrong choice as C would be. Rust is a good choice for situations where, if Rust didn’t exist, C would be a good choice. If you wouldn’t consider using C for a project, don’t use Rust. Use Erlang/Elixir, C#/F#, JavaScript/TypeScript, Swift, Pony, or anything else that targets application programming problems.

                            1. 20

                              There are a lot of safe applications languages with more mature ecosystems, better tooling, and so on

                              I think this is sadly not entirely true. On balance, I would say Rust’s “quality of implementation” story is ahead of mature languages, and that’s exactly what creates this “let’s write CRUD in Rust” pressure.

                              Let’s say I write something where I don’t need Rust, what are my choices?

                              • Erlang/Elexir – dynamic types, the same, if not bigger, level of weirdness in Rust.
                              • F# last time I’ve checked, the build system was pretty horrible (specifying files in a specific order in an XML file)
                              • C# – until recently, it had nullability problems, and was pretty windows-specific. maybe it is a viable option now. My two questions would be: a) how horrible is the build system? b) how hard is it to produce a static binary which I can copy from one linux PC to another without installing any runtime libraries?
                              • JavaScript/TypeScript – npm is a huge time sink. Once deno matures (or Rome conquers the world) it might become a reasonable alternative
                              • Swift – doesn’t really exist outside of Mac? A lot of core things were in mac-specific libraries last time I’ve checkd
                              • Pony – super niche at this point?
                              • Java – nullability, problems with static binaries, horrible build system
                              • Kotlin – Java sans nullability
                              • Go – if you value utmost simplicity more than sum types, that is a splendid choice. I think Go is the single language which is in the same class as Rust when it comes to “quality of implementation”, and, arguably, it’s even better than Rust in this respect. If only it had enums and null-checking…
                              • OCaml – this one is pretty ideal when it comes to the language machinery, but two+ build systems, two standard libraries, etc, make it not a reasonable choice in comparison.

                              Really, it seems that there’s “Go with enums / OCaml with channels and Cargo”-shaped hole in our language-space these days, which is pretty imperfectly covered by existing options. I really wish that such language existed, this would relieve a lot of design pressure from Rust and allow it to focus on systems use-case more.

                              1. 9

                                JavaScript/TypeScript – npm is a huge time sink. Once deno matures (or Rome conquers the world) it might become a reasonable alternative

                                I’d have a hard time finding a productive lead or senior engineer for Node.js complaining that npm is a timesink and the number one thing that makes the language not productive. The teams I work with are incredibly productive with Node (and would have a very hard time onboarding with rust), and most of the time we only have to touch package.json once every few weeks.

                                1. 6

                                  Ocaml 5 has channels along with the new multi core support. And algebraic effects.

                                  1. 4

                                    [… ] this would relieve a lot of design pressure from Rust and allow it to focus on systems use-case more.

                                    Considering that a lot of more serious “system use-cases” (the Linux kernel, Firefox) tend to either fight or ditch Cargo, could it be that its current design is optimized for the wrong use-cases?

                                    1. 3

                                      That’s a great question.

                                      The primary answer is indeed that’s not the use-case Cargo is optimized for, and that was the right thing to do. With the exception of the kernel, all folks who ditch Cargo still make use of crates.io ecosystem. And producing re-usable libraries is the use-case Cargo is very heavily skewed towards. The main shtick of Cargo is the (informal, implementation-defined) specification of Cargo.toml file and .crate archive which provide a very specific interface for re-usable units of code. Because every library on crates.io is build in exactly same inflexible way, re-using the libraries is easy.

                                      The secondary answer is that, while Cargo could be much better (defaults steer you towards poor compile times) for the case of developing a leaf artifact, rather than a library, it’s pretty decent. Stuff like ripgrep, cargo, rust-analyzer, wasmtime are pretty happy with Cargo. There’s a lot of “systems-enough” software which doesn’t need a lot of flexibility in the build process.

                                      But yeah, ultimately, if you build something BIG, you probably have a generic build-system building the thing, and the way to plug Rust there is by plugging rustc, not cargo. Though, yeah, Cargo could’ve been somewhat more transparent to make this easier.

                                      1. 1

                                        The main shtick of Cargo is the (informal, implementation-defined) specification of Cargo.toml file […]

                                        An even bigger shtick is build.rs, a rudimentary build system built with environment variables. For example, in build2 we could map all the concepts of Cargo.toml (even features), but dealing with build.rs is a bleak prospect.

                                        1. 2

                                          Yeah, build.rs is super-problematic. It’s a necessary evil escape hatch to be able to use C code, if you have too.

                                          I think the most promising solution in this space is the metabuild proposal: rather than compiling C “by hand” in build.rs, you specify declarative dependencies on C libraries in Cargo.toml, and then a generic build.rs (eg, the same binary shared between all crates) reads that meta and builds C code. This would provide a generic hook for generic build systems to supply Cargo with C deps.

                                          Sadly, that didn’t go anywhere:

                                          • the design we arrived at required first-class support in Cargo, which didn’t materialize. I feel that an xtask-shaped polyfil would’ve worked better.
                                          • we lacked a brave person to actually submit tonnes of PRs to actually move ecosystem towards declarative native deps.
                                          1. 1

                                            Yeah, in a parallel universe there is a standard build system and package manager for C/C++ with Rust (and other languages) that are built “on top” simply reusing that and getting all the C/C++ libraries for free.

                                    2. 3

                                      I think this overstates the importance of the output being a single static binary. Sometimes that’s a desirable goal, no question. But if you are writing code that will be deployed to a containerized environment, which is the case for a lot of CRUD web apps, installing runtime dependencies (libraries, interpreter, etc.) is a no-op from an operational point of view because the libraries are bundled in the container image just like they’d be bundled in a static binary.

                                      1. 3

                                        But then you’re paying the cost of building container images - which is a huge part of deployment time costs if I understand people’s experience reports correctly.

                                        1. 5

                                          Even with static binaries you’re probably still looking at containerisation for a lot of intermediate things, or runtime requirements, if only because of the state of infrastructure right now. There isn’t really “k8s but without containers”.

                                          1. 2

                                            Not really. Building Docker images with proper caching is pretty quick, within about a couple of minutes in my experience. Far more time will be spent running tests (or in bad CI systems, waiting for an agent to become ready).

                                            1. 1

                                              👍 It only adds seconds on top of iterated builds if done well. Could take an extra minute or two on a fresh machine

                                            2. 1

                                              It’s pretty quick if you choose not to include a whole operating system

                                            3. 2

                                              Yeah, for specific “let’s build a CRUD app” use-case static linking is not that important.

                                              But it often happens that you need to write many different smaller things for different use-cases. If you pick the best tool for the job for every small use-case, you’ll end up with a bloated toolbox. Picking a single language for everything has huge systemic savings.

                                              This is the angle which makes me think that “small artifacts” is an important part of QoI.

                                            4. 3

                                              the same, if not bigger, level of weirdness in Rust.

                                              In the case of Elixir, I’d argue that it actually writes a great deal like Python or Ruby, plus some minor practical guardrails (immutability) and an idiom around message-passing.

                                              The really weird stuff you don’t hit until Advanced/Advanced+ BEAM wizard status.

                                              1. 1

                                                This is your regularly scheduled reminder that D exists. It recently added sumtypes to the standard library!

                                                1. 6
                                                  $ cat main.d
                                                  import std.stdio;
                                                  
                                                  void main(){
                                                      auto r = f();
                                                      writeln("hello, ub");
                                                      writeln(*r);
                                                  }
                                                  
                                                  auto f() {
                                                      int x = 92;
                                                      return g(&x);
                                                  }
                                                  
                                                  auto g(int* x) {
                                                      return x;
                                                  }
                                                  
                                                  $ dmd main.d && ./main
                                                  
                                                  hello, ub
                                                  4722720
                                                  

                                                  If, by default, I can trivially trigger UB, the language loses a lot of points on QoI. I think there’s some safe pragma, or compilation flag, or something to fix this, but it really should be default if we think about language being applicable to CRUD domain.

                                                  1. 2

                                                    This is fixed by enabling the experimental DIP1000 feature via -dip1000:

                                                     11: Error: returning `g(& x)` escapes a reference to local variable `x`
                                                    

                                                    DIP1000 is in testing and will hopefully become standard soon.

                                                    That said, just don’t use pointers. You practically never need them. We have large backend services in production use that never use pointers at all.

                                                2. 1

                                                  Scala. The horsepower and ecosystem of the JVM, plus the type system inspired by much more academic languages. In reality though, all of the above mainstream languages are plenty good enough for almost all line of business work. And where they have shortcomings, certainly Rust does have its own shortcomings as outlined in the OP. So it’s all about tradeoffs, as always.

                                                  1. 1

                                                    tbh, I think the only reason to use Scala over Kotlin is having an already existing large Scala code-base which went all-in on type-level programming (which I think is surprisingly the case for a lot of bank’s internal departments).

                                                    Scala deserves a lot of credit for normalizing functional programming in the industry, it was the harbinger of programming language renaissance, but, as a practical tool, I don’t think it is there.

                                                    1. 1

                                                      As far as I know Kotlin still doesn’t have a general pattern-matching implementation. This is so important for business logic correctness that I find it difficult to imagine managing a serious production codebase without it. I use it on an almost daily basis in my projects.

                                                3. 6

                                                  Use Erlang/Elixir, C#/F#, JavaScript/TypeScript, Swift, Pony, or anything else that targets application programming problems.

                                                  Didn’t Wallaroo move from Pony to Rust around this time last year?

                                                  I remember the corecursive interview with Sean Allen in 2020 talking about his developer experience with Pony “All of us who worked on Wallaroo in the early days have a bit of scar tissue where even though none of us had hit a compiler bug in forever, we were still like, ‘Is that a bug in my code or is that a bug in the compiler?’”

                                                  Having worked at a startup where we used Haskell for an elaborate CRUD app, this sounds about as painful. They sacked the engineering director after I left and I believe they’ve moved to Typescript for everything now.

                                                  I wouldn’t put Pony on that list just like I wouldn’t put Haskell on that list.

                                                  I can also say that my buddy who’s a manager at a big-data company has had everyone under him switch over to Java from Scala.

                                                  So I’d probably put Java and Golang at the top of the list for a CRUD app.

                                                  1. 2

                                                    Great interview, enjoyed reading it.

                                                    Tbh I think the trade off here is how “business logicky” your app is, rather than crud or whatever. At a certain point something like scala really helps you find cascading implications of changes. If you have that kind of app, all else being equal, a rich static type system is going to improve productivity. The more you’re mostly just doing io with code that’s fairly self contained, the easier it is to crank out code with a golang or mypy like type system.

                                                    Similarly the more you need to spend time wrangling for throughput the better your life will be with systems that focus on safety and control of execution.

                                                    Twitter started out on rails, and that makes sense when you don’t need your application to actually be bug free or fast but you do need to just get stuff out there. We continue to have so many different programming systems because there are so many different points in the space that it’s worth optimizing for.

                                                    (For anyone who thinks I’m being dismissive- I spend my days writing golang with only api tests mostly to run sql, and it’s a solid choice for that kind of thing).

                                                1. 1

                                                  Let me leave the following link here in case someone is looking for a pkg-config C library: https://github.com/build2/libpkg-config

                                                  1. 1

                                                    Neat! This is sort of a hash tree fixed at two levels, as I see it (from my cursory reading.)

                                                    Question: I have always stayed away from Boost stuff on the assumption that using any one class from it will drag in a ton of other dependencies, ballooning my build time and possibly code size. Is this still true?

                                                    1. 1

                                                      Unfortunately, yes, it’s still true. In fact, it’s not that you get half of Boost with any library, but you get some dependencies that don’t make any sense. For example, Boost.Graph depends on Boost.Regex (and thus ICU) apparently because Boost.Graph parses some obscure data format using regex.

                                                    1. 15

                                                      Brooks and his MMM had greater influence on me as a software engineer than any other person/book in the field. The chapters on conceptual integrity put into crisp words how I felt about software design but did not have the vocabulary to express my feelings. His books also made me realize that the precision of expression in the spoken language can be as powerful as in programming languages.

                                                      RIP

                                                      1. 1

                                                        For example both languages offer a way to represent a dynamic sequence of elements of the same type stored next to each other. That’s std::vector in C++ or std::Vec in Rust. Both define a vector as a pointer to some memory, a capacity and a length. But what type does the pointer have? How does the data pointed to need to be aligned in memory? What type represents capacity and length? In which sequence are pointer, capacity and length stored?

                                                        I wonder if anyone considered providing a custom C++ standard library implementation that maps directly into the corresponding Rust type representations. Such an implementation could also maintain Rust invariants, e.g., that std::string always contains valid UTF-8.

                                                        1. 7

                                                          Testify committer here. I haven’t done much work on Testify in a couple years other than occasionally boosting PRs, but, a few years ago, I was involved in a bit of cleanup on the project and the codegen was pretty annoying. I don’t recall the exact issues, but basically it would only run correctly in a very specific environment and different environments would produce different results. In some cases there had been PRs merged without re-running the codegen, which obviously caused weird issues. That isn’t to say that I disagree with it, just that the complexity involved is a real consideration.

                                                          1. 3

                                                            In some cases there had been PRs merged without re-running the codegen, which obviously caused weird issues.

                                                            Golden rule of codegen: it should be impossible to check in stale code. Two ways to achieve that:

                                                            • don’t check-in generated code, rely on build system to guarantee it is run as a part of the build
                                                            • check-in generated code, and add a test to verify freshness
                                                            1. 1

                                                              Two ways to achieve that:

                                                              A third way:

                                                              • check-in generated code, and make your build system automatically update it in the source tree as part of the build

                                                              With this approach, the only way to end up with stale generated code is to commit a change without building/testing it. But nobody does that, right?

                                                              1. 2

                                                                With this approach, the only way to end up with stale generated code is to commit a change without building/testing it. But nobody does that, right?

                                                                Well, I totally do commit&push without local testing, relying on CI to run the test asynchronously. So I think if we literally do just what you are suggesting, I’d sneak in stale code. So, the build system should also fail build if the generated code is stale and it’s run on CI.

                                                                1. 1

                                                                  Don’t you at least build locally to make sure there are no syntax errors, etc? If yes, then that would be enough.

                                                                  1. 3

                                                                    Most of the times :) . In general, I feel that “server side” checks work much better for this kinds of things. That is, I in general try to push as much properties to CI, to enforce they actually hold for the master branch.

                                                            2. 1

                                                              My limited experience with code generation makes me think that it’s a good practice to consign it to a dedicated package, that only or mostly contains generated code. Mixed hand-written and generated code will always make it easy (or tempting) to forget to rerun code generation.

                                                            1. 6

                                                              One of the only times linked lists are “undoubtedly the right option” is if you’re writing some sort of intrusive data structure to avoid an allocation by reusing caller stack from another location. Not sure how many run into that often.

                                                              Also in an osdev context, where you have drivers/filters/etc that chain operations to the next installed one, etc. and you want the control flow to be left to each filter/driver/etc in turn rather than making the decision top-down.

                                                              1. 5

                                                                Linked lists are great for stack or queue data structures. They have O(1) insert at the start and ends. Traversal is painful for caches but inserting at one end and removing from either that end or the other end are very fast and does not require allocation. There are a lot of situations where those are useful properties. They also allow constant-time insertion into the middle.

                                                                Doubly linked lists allow constant-time insertion removal from middle without traversal. Array-like data structures are linear complexity for this. Again, this is a useful property for some use cases.

                                                                Linked lists are really bad for cache usage (and TLB pressure) for traversal. If your algorithm requires iterating over the list anywhere near a hot path, they’re probably the wrong data structure.

                                                                1. 1

                                                                  The problem with the use cases you listed can sometimes be the 8-16 bytes of extra memory per item are rather inefficient. For that, you often just use linked list of arrays to amortize the pointer size across multiple items.

                                                                  1. 1

                                                                    You only safe that overhead if the objects are inline in the arrays, but then removal is more expensive because you either defragment the array with copies (cost scales with object size) or you have more expensive search for the next element (lots of data dependent branches to hurt prediction).

                                                                    Lists of arrays are great for things like text, where you want fast iteration and fairly cheap insertion. There, though, the goal isn’t to amortise the cost of the pointer (though that’s a nice benefit), it’s to improve cache locality on traversal. If the list elements are cache-line aligned then you can also vectorise most per-character operations with the widest vector unit available and get good performance. Even in this use case though, you often want random access and so use a richer indexing structure than a list.

                                                                2. 4

                                                                  Could you give a code example of that usecase? I’m a bit dense today.

                                                                  1. 9

                                                                    Not being dense at all; it’s not something most people use outside of low-level dev. Instead of having a list that exists independent of the object(s) and storing the objects in the list, the list is incorporated into the structure of the object (not very OOP!). The clever part is that these objects may be pre-existing in very different contexts/locations (such as the stacks of callers in different threads) and so you now have a way to navigate between all these objects without having dynamically allocated them anywhere.

                                                                    It’s also used as an esoteric concept for writing performant low-level multithreading primitives or frameworks. Thread A wants to acquire a user-space mutex. Instead of the mutex containing a vector of some struct { thread_id, timeout, etc } and need to dynamically allocate, reallocate, etc all that each time a new thread is waiting on the mutex, Thread A reserves space for that structure (with an additional next pointer in it now) on its stack inside the mutex.wait() function but before the actual blocking wait is carried out internally. The mutex just has a single pointer current_waiter that’s either null or the first thread to block waiting for the mutex. Thread A either sets current_waiter to point to the area it reserved on its own stack or traverses the what-we-now-call-intrusive linked list to add itself at the end (no allocations needed).

                                                                    Also, intrusive data structures let you do crazy things like store the same object in multiple lists/trees at once without needing to refcount.

                                                                  2. 2

                                                                    I do this quite often. It’s basically a stack-allocated linked list where nodes reside in different frames of the call stack. Here is one example: https://github.com/build2/libbutl/blob/master/libbutl/diagnostics.hxx#L284

                                                                    Another common case where I tend to use a list instead of a vector is when I need iterator and/or node stability.

                                                                    1. 1

                                                                      (Disclaimer: 30 years systems programming, still going strong…)

                                                                      “One of the only times.. intrusive data structure.. stack re-use”

                                                                      I don’t necessarily agree with this framing. I’d prefer to say, you can make extreme gains by using a linked list, for as long as you well-order your data according to performance requirements .. whereby the stack is hardly relevant as anything but pointer angels pointing to a haven of text segments.

                                                                    1. 5

                                                                      Great read, the last bit especially, which starts with:

                                                                      We don’t officially know how IBM stores these tags.

                                                                      1. 3

                                                                        building the compiler itself used to require 9.6GB of RAM, while now it takes 2.8GB

                                                                        Wow, how can a compiler for a language like ZIP use that much memory?

                                                                        1. 6

                                                                          One reason would be that the entire project is giant compilation unit.

                                                                          1. 3

                                                                            Is it? Then the size of the project would be limited by available memory or CPU architecture. Doesn’t the Zig compiler support separate compilation?

                                                                            1. 1

                                                                              What’s wrong with whole program compilation? http://mlton.org/References.attachments/060916-mlton.pdf

                                                                              1. 8

                                                                                That it requires 9.6GB of RAM?

                                                                                1. 7

                                                                                  Optimizing compilers and linkers use a lot of ram. Because it is generally accepted that developers would prefer compile time be shorter, and generally will happily trade ram for that - it is much easier to double the ram in your machine that it is to double the cpu performance.

                                                                                  This post does say “hey, we’ve made compilation use less ram”, which I’m going to guess was some particular section was using a data structure that had the trade off skewed, or configured with the wrong trade off.

                                                                                  Sure you can point to old compilers that used less ram, but they produced worse code, took longer, or both.

                                                                                  There are plenty of reason to bash Zig, but this just isn’t one of them.

                                                                                  1. 8

                                                                                    I think the real tradeoff is that C compilers used to operate a line at a time, more or less. That warps the language and requires things like forward declarations and the preprocessor.

                                                                                    (The preprocessor also enables separate compilation – parallelization by processes and incremental builds, which is nice.)

                                                                                    But no sane language would make those language concessions now, including Zig.


                                                                                    Still I would like to read a blog post about why self-hosted Zig requires 2.8 GB of RAM. I’m not saying it is too much, but I think it would be instructive.

                                                                                    Especially after the talk about data-oriented programming and Zig’s tokenizing / parsing / AST (which I found useful).

                                                                                    I thought the Zig compiler was around 100K lines of code, not 1M lines of code. So very naively speaking, that would be 28 KB of memory per line, which is ~1000x blowup on the input size.

                                                                                    The code representation doesn’t require that much blowup – it’s 10x at most. So what’s are the expensive algorithms for the other 100x? Type checking, executing comptime, register allocation, etc. ?

                                                                                    That would be a very interesting analysis

                                                                                    1. 9

                                                                                      I was curious too, so I learned how to use massif (turned out to be near-trivial) and collected this data:

                                                                                      https://i.imgur.com/pAUASx4.png

                                                                                      Appears to be mostly LLVM. So another interesting data point would be how much memory is used when Zig builds itself without LLVM involved.

                                                                                      Zig’s non-LLVM x86 backend is not capable of building Zig yet, but I can offer a data point on building the behavior tests, which total about 31,000 lines: peak RSS of 90 MiB

                                                                                      Another data point would be using that contributor’s branch that improves the C backend mentioned in the post- I’m actually able to use it to translate the Zig self-hosted compiler into C code. Peak RSS: 459 MiB Visualization: https://i.imgur.com/ww23lx3.png So here it looks like the culprit is, again, buffering the whole .c file output before writing it. I would expect the upcoming x86 backend to have an even better memory profile than this since it does not buffer everything in memory.

                                                                                      1. 4

                                                                                        Ah thanks, well this makes me realize that I read the title / first two paragraphs and assumed that pure Zig code was taking 2.8 GB :-/ Judging by other comments, I was probably not the only one who thought that …

                                                                                        This makes more sense – from what little I know about LLVM, there are lots of hard algorithms that are quadratic or exponential in time or space, and heuristics to “give up” when passes use too many resources. I’d definitely expect the hot patching Zig compiler (a very cool idea) to use less memory, since the goal is to generate code fast, not fast code


                                                                                        I also sympathize with the 2 years of “under the hood” work – it’s a similar story for https://www.oilshell.org right now !

                                                                                        1. 4

                                                                                          Yeah it’s a bit awkward to disambiguate between these things:

                                                                                          • frontend in C++, backend in LLVM (“old bootstrap compiler”)
                                                                                          • frontend in Zig, backend in LLVM (“self-hosted compiler”)
                                                                                          • frontend in Zig, backend in Zig (“fully self-hosted compiler”)
                                                                                          • frontend in C, backend in C, outputs C (“new bootstrap compiler”)

                                                                                          All of these are relevant codebases at this point in time. What would you call them?

                                                                                          1. 3

                                                                                            I don’t have that strong an opinion, but I would say “self-hosted front end” makes sense – i.e. “Zig’s front end is self hosted”, but I wouldn’t yet say the “Zig compiler is self-hosted”

                                                                                            I don’t think of the last one as a “bootstrap” compiler. The term “bootstrapping” is overloaded, but I think of that as the first one only – the thing you wrote before you had Zig to write things in!

                                                                                            If the purpose of the last one is to run on architectures without LLVM, then maybe “generated compatible compiler” or “generated compiler in C” ?

                                                                                            That said, I probably don’t understand the system enough to suggest good names. I can see why the last one would be used for “bootstrapping” a new platform. (“Turning up” instead ?)

                                                                                            stage{0,1,2,3} might be OK too – what I do is link all terms to the “glossary”

                                                                                            https://www.oilshell.org/cross-ref.html

                                                                                            Although that doesn’t necessarily help people writing on other sites! What I really do is explain things over and over again, while trying to converge on stable terms with explicit definitions … but the terms change as the code changes, and the project’s strategy changes, so I understand the problem :)


                                                                                            edit: I think the first one “bootstraps the Zig language” and the second one “bootstraps a new platform with Zig”, which are related but different things. I think having 2 different words for those could reduce confusion, but you would probably have to invent something (which is work, but IMO fun)

                                                                                            1. 1

                                                                                              frontend in C, backend in C, outputs C (“new bootstrap compiler”)

                                                                                              Interesting, didn’t realize y’all were doing a new bootstrap compiler too. Just wondering why it doesn’t make sense to have the fully self-hosted compiler compile to C for bootstrapping?

                                                                                              1. 2

                                                                                                I’m fine with doing that for a little while but it does not actually solve the bootstrapping problem. Bootstrapping means starting with source, not an output file (even if that output file happens to have a .c extension).

                                                                                                1. 1

                                                                                                  IIRC, that’s the plan

                                                                                          2. 2

                                                                                            Still I would like to read a blog post about why self-hosted Zig requires 2.8 GB of RAM ….

                                                                                            That brings it to the point, thanks; I’ve read that the compiler is 200kLOC, but 500x blowup is still very, very much; assuming that the backend is still LLVM, we can assume that the frontend was responsible for the additional 6.8 GB, which makes it even more mysterious.

                                                                                          3. 3

                                                                                            it is much easier to double the ram in your machine that it is to double the cpu performance.

                                                                                            But doubling RAM is only half of the story: you also need adequate memory bandwidth. You can see the effect of this in how poorly (non-Pro) Threadrippers scale for C++ compilation.

                                                                                            1. 1

                                                                                              Yes, but it’s still easier to double the ram than double the cpu runtime performance - as the last decade I guess has shown the bulk of the big compute increases are from increasing the number of cores, which in general does not seem to benefit compilation of individual translation units.

                                                                                              Obviously you would ideally have faster and uses less ram, but that’s true of anything with a trade off: we’d rather there not be one :)

                                                                                              So I would rather something use more ram and get things done faster than skimp on memory out of some arbitrary misplaced fear of using the available system resources.

                                                                                              A lot of my work in JSC was directly memory vs page load time, and people seem to have found that to be the correct trade off.

                                                                                              1. 1

                                                                                                TU? plt?

                                                                                                1. 2

                                                                                                  Updated the comment to remove the abbreviations sorry.

                                                                                                  TU = translation unit, eg the piece of code a compiler is “translating” to assembly or what have you, for example a single .c file (including all the included headers, etc) would be a TU

                                                                                                  Plt: super ambiguous here sorry, here it is page load time, but contextually you could reasonably have thought programming language theory, though that would make the sentence even more confusing :)

                                                                                                  1. 1

                                                                                                    Translation Unit

                                                                                                    Programming Language Theory

                                                                                                    1. 1

                                                                                                      Wrong in the latter! :)

                                                                                                      Plt here is page load time :)

                                                                                                      1. 1

                                                                                                        Aha! Thank you for the correction

                                                                                                        1. 2

                                                                                                          Correction is strong - I used plt in a conversation about programming languages, and was not referring to programming languages, what could go wrong? :D

                                                                                              2. 2

                                                                                                Optimizing compilers and linkers use a lot of ram

                                                                                                I started building compilers three decades ago and have a formal education to do so and there are a lot of compilers still developed today which use less RAM for the same source code size, are faster and don’t generate worse code. Therefore it’s a fair question what the Zig compiler does differently.

                                                                                                1. 2

                                                                                                  there are a lot of compilers still developed today which use less RAM for the same source code size, are faster and don’t generate worse code.

                                                                                                  Now I’m curious! Which compilers were you thinking of here?

                                                                                                  1. 1

                                                                                                    E.g. all C and C++ compilers I worked with; my favorite is still GCC 4.8.

                                                                                                    1. 1

                                                                                                      Interesting…given that apparently most of Zig’s memory usage turned out to be LLVM [0], I’d be really surprised if clang used less RAM and ran faster for the same size C++ code, but I have very little firsthand experience with either clang or sizable C++ code bases, so maybe that’s just my misimpression of clang and C++!

                                                                                                      edit: just realized it’s also possible that clang isn’t one of the C++ compilers you’ve worked with.

                                                                                                      [0] https://lobste.rs/s/csax21/zig_is_self_hosted_now_what_s_next#c_bspzkq

                                                                                                      1. 3

                                                                                                        To provide some figures, I built my most recent LeanQt release (https://github.com/rochus-keller/LeanQt) with different compilers on x86 and x86_64.

                                                                                                        Here are the cloc results:

                                                                                                        LeanQt-2022-10-21/core $ cloc .
                                                                                                             491 text files.
                                                                                                             491 unique files.                                          
                                                                                                               3 files ignored.
                                                                                                        
                                                                                                        http://cloc.sourceforge.net v 1.60  T=3.81 s (128.1 files/s, 76031.5 lines/s)
                                                                                                        -------------------------------------------------------------------------------
                                                                                                        Language                     files          blank        comment           code
                                                                                                        -------------------------------------------------------------------------------
                                                                                                        C++                            209          32448          64890          98328
                                                                                                        C/C++ Header                   265           9203          11711          68493
                                                                                                        Objective C++                   10            387            484           1961
                                                                                                        IDL                              1             12              0            963
                                                                                                        Assembly                         3            106             67            682
                                                                                                        -------------------------------------------------------------------------------
                                                                                                        SUM:                           488          42156          77152         170427
                                                                                                        -------------------------------------------------------------------------------
                                                                                                        

                                                                                                        Here is the result of gcc version 4.8.2, Target: i686-linux-gnu

                                                                                                        /usr/bin/time -v ./lua build.lua .. -P HAVE_CORE_ALL
                                                                                                        	Maximum resident set size (kbytes): 119912
                                                                                                        

                                                                                                        Here is the result of gcc version 5.4.0, Target: x86_64-linux-gnu

                                                                                                        /usr/bin/time -v ./lua build.lua .. -P HAVE_CORE_ALL
                                                                                                        	Maximum resident set size (kbytes): 286628
                                                                                                        

                                                                                                        Here is the result of Apple LLVM version 7.3.0 (clang-703.0.31), Target: x86_64-apple-darwin15.6.0

                                                                                                        /usr/bin/time -l ./lua build.lua ../LeanQt -P HAVE_CORE_ALL
                                                                                                         117989376  maximum resident set size
                                                                                                        

                                                                                                        Conclusion: A C++ code base comparable in size with the Zig compiler requires less than 300 MB RAM with all tested GCC and Clang versions; Clang x86_64 requires even less than half of the RAM than GCC x86_64 (118 MB vs. 287 MB).

                                                                                                        1. 2

                                                                                                          Wow, not at all what I would have guessed - thank you so much for sharing!

                                                                                                        2. 1

                                                                                                          I’m also using Clang/LLVM; the most recent version on my M1 Mac is 13.x, though my favorite version is still 4.x.

                                                                                                    2. 1

                                                                                                      Sure, I would guess that there are places where they have chosen clearer/more easily understood architecture and algorithms because modern hardware changes the trade offs.

                                                                                                      Because even on individual TUs in clang can easily hit gigs of ram at a time, LTO modes easily make it insane.

                                                                                                      You could argue “lazy devs aren’t doing what we had to do in the past” but that would simply mean you should turn around and say “why bother making CPUs faster, or making systems with more ram”.

                                                                                                      I don’t have three decades of compiler experience unless you consider my masters or work on the Gyro patch for Rotor, which you could argue is “compiler work”, but as none of this was real production level code I wouldn’t in this context. So let’s say I have somewhere in the 10-15 year range of production and shipping to consumers and developers, but I think that my experience is sufficient here.

                                                                                                      1. 1

                                                                                                        unless you consider my masters …

                                                                                                        That was not the point. The point was that I came across a lot of compilers and languages, even in a time when 100 MB was an incredible lot of RAM, and that I cannot explain why a language like Zig, which is not the most complex language there is (certainly less complex than C++) requires ten to hundred times more memory than e.g. the C++ compilers I ever had to do with. Even if we can make educated guesses, they are still guesses.

                                                                                                        1. 1

                                                                                                          That’s literally the first part of my answer:

                                                                                                          Sure, I would guess that there are places where they have chosen clearer/more easily understood architecture and algorithms because modern hardware changes the trade offs.

                                                                                                          Increasing the available resources (ram, cpu time) is an enabler - if you no longer have to worry about every single byte, or every single cycle, in order to make a robust and usable compiler. Presumably if people start making large scale projects in Zig, the trade offs between simplicity and ease of development vs. performance will change, I assume the 9+ -> 2.7gb reduction was the result of something along those lines.

                                                                                                  2. 2

                                                                                                    It now takes 2.8GB of RAM so I’m not sure what your point is?

                                                                                                    1. 4

                                                                                                      That brings us back to my original question: Doesn’t the Zig compiler support separate compilation? 2.8 GB is still too much if you e.g. want to compile on an x86 Linux machine.

                                                                                                      1. 1

                                                                                                        Not that this is the only answer, but you can easily cross compile with Zig. So you need not compile on an x86 machine.

                                                                                                        1. 1

                                                                                                          Separate compilation doesn’t get you a whole lot - if you have standard 1 to 1 of TU to a process, you are now using more memory concurrently than a single process would.

                                                                                                          The reality is that a lot of compile time performance is achieved by trading off against memory use (that’s how tu/process works), And a lot of the more advanced optimization algorithms for runtime have very large working sets - which you can carefully reduce the size of, but frequently at the cost of compile time again.

                                                                                                          For many compiler devs the implementation complexity required for fast compilation in a 32bit address space is not worth it on its own, let alone the opportunity cost of doing that instead of something else.

                                                                                                          1. 1

                                                                                                            Separate compilation doesn’t get you a whole lot

                                                                                                            Is this the confirmation that Zig doesn’t support separate compilation?

                                                                                                            the implementation complexity required for fast compilation in a 32bit address space is not worth it on its own

                                                                                                            I can easily compile the Linaro ARM GCC which is even bigger than the 200 kLOC of Zig on an x86 Linux machine and it doesn’t use more than a few 100 MB RAM to work. Zig is supposed to be a “better C”, isn’t it?

                                                                                                            1. 2

                                                                                                              is this confirmation…

                                                                                                              No idea, I don’t like Zig as I disagree with a bunch of their core language design decisions, so haven’t investigated any implementation details :)

                                                                                                              I can compile…

                                                                                                              Again no idea about the zig compilation, but all 32bit compilers explicitly drop a bunch of compile time performance optimizations due to address space constraints, the extreme case being compilers from a few decades ago that did essentially line by line compilation because anything more burned too much ram - it’s why [Obj-]C[++] all require forward decls (though in c++ templates also don’t help :) )

                                                                                                              1. 3

                                                                                                                Here is some x86 and x86_64 data with compiler versions between 2013 and 2019: https://lobste.rs/s/csax21/zig_is_self_hosted_now_what_s_next#c_2lc2dk. Much less than the available memory is used, even on 64 bit systems.

                                                                                                                1. 2

                                                                                                                  Thanks for all that work!

                                                                                                                  My opinion is very much that as a more recent language Zig took the imo reasonable approach of using standard data types and algorithms, rather than custom everything that llvm, clang, gcc, etc have.

                                                                                                                  Because of your work I’m now curious just how much memory is saved in llvm+clang (I know next to nothing about the gcc code base) by the large amounts of effort in keeping memory use down (llvm originated in an era with much less ram, and clang less so, gcc was literally decades earlier so presumably also does fairly complicated stuff to keep size down).

                                                                                                                  But the big thing that’s different is that in earlier times a lot of core architectural decisions in the older compilers that result in much more complicated data types than I suspect the Zig implementation does. There are numerous different versions of the common core data types in llvm specifically to keep memory use down, other things like how the IR, ASTs, etc get kept around make the types themselves obnoxious but also impacts the architecture as removing info from some types means you need to have fast ways to get that info again.

                                                                                                                  Why would a new language take on that complexity - especially if it’s trying to be welcoming to new devs (imagine if you were introduced to C or C++ by some absurd macro&template monstrosity instead of clean easy to comprehend code)

                                                                                                    2. 2

                                                                                                      The presentation you linked mentions compiling a 100k program with less than 1 GB RAM. If zig is about 150 KLOC (like I think I saw elsewhere in the thread) then I think it lends to Rochus’s point that 9 GB was pretty heavy.

                                                                                                      Of course, the presentation is from 2006 and the MLton authors were probably more concerned with 32 bit builds so they might have made more efforts to keep build sizes low. (Which is to say it was different because things were different. Woo-hoo!)

                                                                                              1. 9

                                                                                                The main goal of this first iteration is to enable simple usage of dependencies to start building a package ecosystem, and to make sure that we can easily package C/C++ projects, not just Zig.

                                                                                                After having gone through this process (i.e., replacing a hodge-podge of build systems with build2) in 300+ C/C++ packages, one especially nasty thing about quite a few of them is the dynamic probing (i.e., compile/link tests) of the target with Autoconf checks (or their CMake equivalent) in order to generate the config.h file. Our solution is the libbuild2-autoconf build system module. I wonder what’s Zig’s plan?

                                                                                                1. 9

                                                                                                  Wow - I’m checking out libbuild2-autoconf now. This is impressive, I can only imagine the pain you had to go through to make this.

                                                                                                  From the look of it, our strategy will be similar to your pragmatic approach. I’d be interested in comparing notes and collaborating wherever it makes sense to!

                                                                                                  If you want something to look at, here’s an experimental SDL prepared as a zig package. Essentially, it’s a fork of upstream, build system replaced with build.zig. As you noted, the config.h is problematic, and in here I didn’t really solve it in a satisfying way - I prebuilt it for a few targets and then manually tweaked it. It also doesn’t solve dynamic linking against X11 on Linux. So yeah we got some problems to solve.

                                                                                                  1. 4

                                                                                                    Sure, we would be happy to collaborate. I suppose you could easily reuse our Autoconf checks if you are happy with the overall approach. And we sure would be glad to reuse any that you implement.

                                                                                                    A more radical idea would be to reuse the build2 build system (which is available as a C++ library) in Zig. Specifically, you could try replacing the “engine” that’s inside build.zig with it. This will not only give you access to the Autoconf functionality, but also to the 300+ C/C++ packages I mentioned above.

                                                                                                    I have even more radical ideas, but this is probably already pusing it ;-).

                                                                                                1. 4

                                                                                                  An interesting property of syntax similar to lisp or XML is that because most of the nodes in the syntax are wrapped in a container tag you can more easily expand the syntax to accommodate for extra attributes and metadata (e.g. you can have extra attributes on XML tags, or child elements within an element to encapsulate metadata related to that specific part of the program).

                                                                                                  The super lean and clean syntax that often lacks wrapper blocks reads nicely and looks pleasant but the downside is that it leaves no room for associating any extra additional information or metadata to that specific part.

                                                                                                  This comes down to somewhat of a fundamental trade-off in syntax design. A verbose and noisy syntax is more difficult to read and looks intimidating but it can capture information more easily which can lead to better programs (i.e. more information and metadata for the compiler to work with and warn you about problems, etc…).

                                                                                                  A lean and clean syntax looks delightful and reads easily but it becomes somewhat hostile to the capturing or representation of extra additional information because there’s no way in the syntax to include any kind of child element or extra key-value attributes.

                                                                                                  You can see people try to break out of this limitation by inventing things like special tags in code comments so a documentation can be auto-generated from the code. And they have to resort to hacks such as leaving comments above a particular function with special “@return”, “@param” keywords.

                                                                                                  Whereas with an XML-like syntax you could more naturally capture that information with dedicated tags within the code. Even though it doesn’t look particularly nice to read.

                                                                                                  1. 3

                                                                                                    Perhaps the middle ground is to systematically reserve extension points in the syntax. For example, in build2 every entity (target, prerequisite, variable, value, etc) can be preceded with attributes enclosed in [ ]. Compare:

                                                                                                    [string, visibility=project] var = [null]
                                                                                                    

                                                                                                    To:

                                                                                                    <binding>
                                                                                                      <variable type="string", visibility="project">var</variable>
                                                                                                      <value null="true"/>
                                                                                                    </binding>
                                                                                                    
                                                                                                    1. 6

                                                                                                      Yes that also seems great.

                                                                                                      I think a huge portion of problems in programming languages eventually boil down to this limit of expressing extra information in the syntax.

                                                                                                      This results in a limited number of attributes getting “first class support” in the syntax and then everything else gets neglected.

                                                                                                      For example when it comes to name bindings or variable declarations most languages end up with something like “const” for constants, or “mut” to describe a mutable binding, and/or other keywords such as “private”, “public”, “protected”.

                                                                                                      Then you are somewhat locked out of extending that syntax or expressing anything more elaborate than those basic ideas.

                                                                                                      If the syntax is “metadata-friendly” it opens the door to expressing lots of extra useful information that can be used by the compiler or other tooling such as for automatic documentation generation.

                                                                                                      Just to name a few examples let me mention some of my wishlist items.

                                                                                                      • I’d like a way to tag the classes/functions in the code and then be able to search for them using those tags. Very useful in a large codebase.

                                                                                                      • Comments and annotations as a first-class feature in the syntax

                                                                                                      • Pre-conditions or post-conditions before/after blocks/functions to act as guards/assertions

                                                                                                      • Ability to explicitly link a test to the thing that it is supposed to be testing, and then being able to query that list. For example the ability to view “all functions tagged with security that have no linked tests associated to them”.

                                                                                                      Lots of possibilities!

                                                                                                      1. 3

                                                                                                        Maybe join me on matrix? Datalisp.is work towards this end but it’s boring to work alone.

                                                                                                        1. 2

                                                                                                          Thanks I’ll check it out

                                                                                                      2. 1

                                                                                                        PowerShell does this too! It’s really nice.