The immediate effects are:

• Some part of Rust community will feel strongly and vocally about misusing unsafe (this is a serious misuse of unsafe according to the current culture).
• There’s probably some bugs where invalid input makes n2 crash, rather than emit an error. Eyeballing, I think this truncate could cause a crash? It asserts that the truncation point is the utf-8 char boundary, which is not something n2 enforces.
• It is unlikely, but possible that there’s some sort of really bad bug where untrusted input would cause an RCE or something.

But, more generally, it feels like this whole thing is really just fighting the language? There’s a very specific, intentional dichotomy between utf-8 – String, everything else – Vec<u8>. It’s not about “why don’t you do something different”, but rather about “why do you do this”. The default action here would be to use Vec<u8> / [u8] for everything. I don’t think you need bstr. It will make certain things nicer, but a vector of bytes is quite capable and convenient type in its own right.

That is if you want to use the raw-bytes model. There’s an argument that you can’t escape encoding (because, on Windows, the build.ninja will be in utf-8/ascii, and the paths are always UTF-16, so something will need to encode even for ascii case). From this perspective, it makes sense to use String for build.ninja, OsString or Vec<u8> for parts that represent paths/arguments (I think OsString is the morally correct choice here, but its internal representation is insufficiently transparent to do things like path simplification easily, so you might want to roll your own on top of Vec<8> here) and Vec<8> for everything that might be encoded (like compiler’s output a-la /showIncludes), with a clean layer that decodes Vec<8> into OsString.

(for completeness, there’s also a school of thought that says that, instead of going out of the way to represent non-utf8 inputs, build tooling should just mandate that everything is utf-8, and error early for everything that’s not, forcing everyone to make their builds non-weird. That is, you’ll still parse WTF-16 you get from msvc, but you error if it can’t be decoded into valid utf-8. https://docs.rs/camino/latest/camino/ is a library for that)

But it does mean there’s a bunch of kind of needless unsafes in the code, and some of them are possibly actually doing something bad.

There are Path and PathBuf types in the standard library for this and it’s not clear why they’ve chosen something more complicated and error prone.

People don’t write code like this. This will scare away potential contributors. If I was looking for Rust job and I knew that in my potential employer codebase &str is not necessary UTF-8, I will probably rejected that job.

Also, you lose benefits of Rust strong type system. Ideally, types should represent sets of allowed values as accurately as possible. If I would write something like n2, I would use different type for any kind of string: one for proper UTF-8, one for arbitrary bytestrings, one for WTF-8, etc, this would help me write correct code. See also https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ .

(Fun fact: when writing this comment, I thought that putting invalid UTF-8 to str is undefined behavior, and thus compiler is allowed to do anything. I did dig to Rust reference in attempt to find a proof. And… it turned out that this is not undefined behavior. :) So, yes, today I became slightly smarter. :) Thank you. :) But I still think that nobody should store arbitrary bytestrings in str/String. Yes, this is not immediate undefined behavior, but standard library docs still warn ( https://doc.rust-lang.org/std/primitive.str.html#invariant ) that a lot of code both in standard library and outside of it assume that str is valid UTF-8.)

Also, such fundamental tools (sitting very deeply in dependency chain) as n2 should not be written in Rust. I love Rust. Today I write everything in Rust. But if I wrote something like n2, I would choose C. (And if I did choose Rust, then I would explicitly scare away people from using the tool for core Linux packages.)

Reason is simple: Rust’s compiler is big and complex. And it is written in Rust itself. This complicates building Rust programs for unusual architectures, even if Rust compiler is already ported to that architecture.

Here is post, which proves my point: https://drewdevault.com/2022/01/15/2022-01-15-The-RISC-V-experience.html . The author built Alpine Linux for his personal riscv computer. And he had to exclude Firefox, because parts of it are written in Rust:

I was able to successfully set up almost all of my standard workstation loadout on it [RISC-V box], with some notable exceptions. Firefox is the most painful omission — bootstrapping Rust is an utter nightmare and no one has managed to do it for Alpine Linux riscv64 yet (despite many attempts and lots of hours wasted), so anything which depends on it does not work. librsvg is problematic for the same reason; I had to patch a number of things to be configured without it.

Then in footnote:

Incidentally, my new language [Hare] can be fully bootstrapped on this machine in 272 seconds, including building and running the test suite. For comparison, it takes about 10 seconds on my x86_64 workstation. Building LLVM on this machine, let alone Rust, takes upwards of 12+ hours. You can cross-compile it, but this is difficult and it still takes ages, and it’s so complicated and brittle that you’re going to waste a huge amount of time troubleshooting between every attempt

As well as I understand, Rust compiler was already ported to riscv at that point, but still building rustc (and everything dependent on it, such as Firefox) happened to be too difficult for Drew.

If (let’s imagine this) n2 becomes widespread tool and many core projects (sitting deeply in dependency chain) start to use it, then building Alpine with Firefox for riscv, etc, becomes even more difficult.

Here is another writing by Drew: https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-replacement.html . One notable fragment:

C is the most portable programming language. Rust actually has a pretty admirable selection of supported targets for a new language (thanks mostly to LLVM), but it pales in comparison to C, which runs on almost everything. A new CPU architecture or operating system can barely be considered to exist until it has a C compiler. And once it does, it unlocks access to a vast repository of software written in C. Many other programming languages, such as Ruby and Python, are implemented in C and you get those for free too.

Please, make no mistake: I love Rust! I write everything in it. And I disagree with many things in this Drew’s article (“Rust is not a good C replacement”). I think that Rust is good C replacement. But I still agree with Drew in one thing: Rust currently is not as portable as C. And writing core tools, such as build systems, in Rust, is unacceptable, because this creates a lot of problems for system creators, for system programmers, etc.

Look at this list: https://bootstrap.debian.net/essential.html . This is list of transitive build dependencies of Debian package “bash” (or “coreutils”, or “libc”, the list will be same no matter where you start). If you port Debian to new architecture, then you should eventually port all packages in the list, otherwise the port cannot be considered “complete”. Every tool in that list transitively depends on every other! This means that author of any tool in that list can easily perform supply chain attack (remember https://en.wikipedia.org/wiki/XZ_Utils_backdoor ) on all Debian packages. I think that this list should be kept as small as possible. To easy porting of Debian to new architectures and to prevent supply chain attacks. Alas, the list grows beyond any control and becomes bigger, not smaller. (And other Linux distros, of course, have similar lists, through they not always actually generate them and put to some web site.) Unfortunately, the list contains rust, too. This is very unfortunate. I strongly believe that rustc should be removed from that list. And yes, this is possible (if someone makes some heroic efforts.) But if your n2 becomes widespread and people start to use it for core Linux/Debian packages, then removing rustc from that list will become harder.

So, again: I think this is totally OK to write n2 in Rust as long as you write warning: “Please, don’t use this build system for core Linux packages sitting deeply in dependency chain”.

(And of course, all these is just advice, of course, I don’t order you to do anything.)

And yes, I think ninja and n2 are both small and nice tools. Thanks for creating them!

Representing and operating on the structured data for the build graph nodes feels a lot more natural and extensible than operating primarily on files like make/ninja. I missed having access to Bazel/buck2-style “providers” for arbitrary structured metadata, and being able to process the metadata directly in the build language, rather than having to shell out.

• [tangential] An interesting difference between build2 targets and Bazel/buck2-style providers seems to be that the former are inheritance-/subtyping-based, while the latter are more interface-/protocol-based.

Of course, one question is: when does Ninja cease to be an “assembler” rather than a “compiler”, as per its design description?

• On one hand, the build metadata doesn’t really ‘need’ to live in the Ninja build graph, rather than the generator.
• On the other hand, there are meaningful performance concessions when you have to represent every node as a file (and possibly correctness implications too, when you consider mtime issues, filesystem issues. etc.).
• I wonder if there’s a minimally-invasive design change that could lift n2 to operate on more abstract nodes?
  • Of course, it may turn out that existing Ninja files wouldn’t be able to take meaningful advantage of it without a rewrite.
  • But maybe it would address certain issues around mtimes or phony rules?

Some additional context for those unfamiliar with Ninja:

• Ninja has rules which can be parametrized, but perhaps the main difference is that the data flows in only one direction?
  • The instantiation of a rule can pass in information (limited to string key-value pairs), but a rule can’t easily ask for information about its inputs.
  • Since Ninja is intended to be generated, it doesn’t matter from an expressiveness standpoint, because you can arrange for rules to be instantiated with all the information they would need.
• IIRC I still found the lack of structure inconvenient when writing/maintaining the generator program.
  • There was a lot of dumping and parsing important build metadata into files as inputs to build rules, and then the rule implementation would have to call back into some script at runtime to interpret the input blob.
  • You might need to forward metadata across many different kinds of rules. I think this means that all rules potentially need to know about all other rules, or you somehow otherwise need to account for this in your generator.
  • It was somewhat painful to implement and reason about the information flow at different evaluation times, although this is just a general compiler problem.

I think this portion in mtime comparison considered harmful (linked from the original article) held true in practice:

Checksumming every input file before building is very slow. If you’re considering whether to rebuild foo.a, and foo.a depends on *.o, and each.o depends on each.c, then you have to checksum the full content of every .c file every time you consider making an incremental build of foo. In large projects, this could be thousands, or tens of thousands of files, each of which we have to open(), read(), checksum, and close(), possibly over a network filesystem. For small projects this is fine, but for large projects, this sucks a lot.

blaze/bazel come from a world where source files are stored in a virtual filesystem, which happens to have the ability to tell you a precalculated checksum for every source file (except the ones you’ve changed locally). If you only have to checksum your locally-changed files on each build, that’ll be very fast. But you need filesystem support to make this possible, and we can’t assume that everywhere.

You would expect hash computation/comparison to be strictly slower than mtime computation/comparison even in the best case:

• In the case of multi-GB artifacts, hash computation could be noticeably slow even on an individual basis.
  • On some ontological basis, you could argue that big artifacts are most likely to be on a critical path, under the hypothesis that smaller artifacts earlier in the build generally combine to become bigger artifacts later in the build.
  • If true, doing hash computation feels painful under the above circumstances because 1) it’s on the critical path and 2) it’s for a bigger artifact and linear in said artifact’s size.
• It’s extra expensive if you have to use a networked filesystem to read the file contents.
• For a large enough project, you might even consider implementing a cache from file metadata to content hash, like mentioned in the “redo: mtime dependencies done right” section of the linked article.
  • It probably gets messy if you want to cache networked file hashes in the same way. I don’t know what kind of metadata is reliable in networked contexts.
  • Also, I see now that they mention a “sequence number” for targets, which is perhaps similar to my original comment of adding a sequence number instead of using mtimes directly.

I suspect that, due to the Ninja architecture, the build and hash computation happened via deep constructive traces, rather than shallow ones:

• Therefore, you couldn’t start the build before computing all the hashes. This was already discussed under “Single pass” in the original article.
• But, even if you had shallow traces, you would still have to hash large artifacts and many artifacts as per above.
• Hash comparison is also fundamentally more expensive than numeric mtime comparison, which eventually starts to add up.
  • I expect this is more of a data locality issue than a computation issue. Do you store the hashes directly in the core “node” data structure, or somewhere else? If you store them directly, then you can have fewer nodes in the cache; if you store them indirectly, then you have to traverse the indirection when invalidating the graph.
  • It’s also just higher peak memory usage, which may or may not be substantial in practice.