I can’t help but wondering why obviously bad edge-cases like this come to exist in the first case. The surface level reason is ‘the scoping model is the problem’, but how does one not understand that such a scoping model is an obvious footgun to begin with (even if the specific issue mentioned here is missed)? C’mon folks, ML and many others got these things right in the 70s. All you needed to do was copy ML. Don’t make obviously terrible language design decisions.
Maybe I’m being unnecessarily harsh: the last decade has brought brilliant progress in language design and perhaps there’s a big hindsight bias here that made this issue less clear during Go’s inception. Still: ideas like ‘hey, maybe don’t let locals outlive the scope in which they were defined’ seem obviously sensible and are generally easier to implement when doing compiler development anyway.
Go chose to have append, modelled after libc realloc. It has pervasive multi-threading and non-thread-safe built-in map. So in general, Go is not that focused on preventing footguns.
This boilerplatey reassignment is crucial. If you forget about the reassignment, or assign to a variable that is a temporary/scoped copy of a variable, rather than the place that owns the slice:
append(slice, item)
then it will sometimes seem to work, until it won’t, depending on capacity.
It’s a very low-level design that lays bare the reallocation happening. It’s fine to have it in the language for advanced uses, but I’d expect language to have an everyday foolproof alternative that ensures the slice’s data pointer is always updated.
And the other shoe is that append looks like it copies the slice so some folks will write s2 = append(s1, item) to create a copy-with-append, this is always an error because if s1 has sufficient capacity they will now be sharing backing buffer, and will overwrite one another’s data.
I find your view fairly harsh, but I can see why you and others may not, and the degree to which it’s “unnecessary” is debatable. Perhaps your view is fair and justified.
My opinion on, “why obviously bad edge-cases like this come to exist in the first case” is simple. People make mistakes, and language designers are people. And while this mistake may seem obvious with hindsight, I don’t think it’s necessarily obvious in the absence of it.
‘hey, maybe don’t let locals outlive the scope in which they were defined’
I am no language design expert so maybe I am missing something. But if we don’t let locals outlive the scope in which they were defined, how would closures work at all? Closures may be called after the scope has exited and the closures still need the variables in the scope that has exited to work correctly, don’t they? What am I missing?
‘Scope’, when it comes to language semantics, doesn’t mean ‘lifetime’.
It’s fine for values to outlive the lifetime of their scope via closures (provided you have safeguards in place to prevent dangling pointers such as a borrow checker, GC, or whatever)
Scopes are generally considered to be ‘naming contexts’: regions of code in which a particular name corresponds to a specific value (or is accessible at all).
I think it’s expected, by most programmers, that the start of a loop body denotes a new scope on every iteration, and that elements from the last iteration probably shouldn’t carry over unless passed out into an enclosing scope as a side-effect (i.e: mutability, or a similar such mechanism).
That’s not to say that the same memory cannot be reused by each iteration, but that previous state should not be observable.
If the loop variable itself is declared outside the loop body and its value persists between loop iterations (like i++) then it makes sense to me that it is a single variable in a parent scope separate from the loop body.
Not everyone who makes a programming language has experience with every other programming language—not everyone who makes a programming language is even a programming language enthusiast. The Go folks clearly built a language aimed at scratching their own itches and they mostly got it right. Most languages fail to do the most important stuff well (at least as it pertains to software development anyway) because they fixate on minutiae like this. I guess I’m glad Go fixed this, but it’s not like I’d base my choice of language on it.
This is literally an attempt to prevent people from breaking their installs, by shutting down one of the most common ways that happens (installing packages into the “system” Python).
I’m honestly not even sure at this point what would make people happy – if something doesn’t get fixed, people complain it’s broken, and if it does get fixed people complain that it’s a change from previous behavior.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
The Python packaging ecosystem has standard tools that work quite reliably. The main problem is people who refuse to use them, build their own rickety Jenga tower of stuff as an alternative, and then blame “Python packaging” when it doesn’t work.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
For years now, pip has tried to detect this kind of situation and issue a warning reminding you that installing into the global “system” Python is a bad idea. And every best-practices guide for how to package and containerize Python that I’m aware of has recommended avoiding installing into the “system” Python and instead creating a venv.
But finally, distros who are tired of getting bug reports from people who did this and broke their system are just enforcing a blunt “nope, you can’t do that”. As I see it, you’ve had ample warning and ample time to learn about and adopt the clear recommended best practice (which is not some sort of constantly-changing thing – “use a venv” has been the advice for years and years now). I don’t see how it’s Python’s fault that you refused up until now.
That dude who founded Sentry just released a new one like 2 weeks ago as well and my Twitter is full of people all hyped about it.
I’m not even trying to say that this fix from Debian totally disabling pip is a bad thing. I’m just complaining about how hilariously messed up the whole Python packaging ecosystem is and what a pain it is to work with it.
There’s literally a website dedicated to listing the dozens of different options for just package management
Which doesn’t contradict what I said. There are in fact standard reliable tools: setuptools as the package build backend, pip as the package installer/uninstaller, and venv as the isolation mechanism.
And like I said, a huge percentage of perceived issues with “Python packaging” are people deciding to go off and build their own thing instead. At which point the pain they deal with is entirely their fault and nobody else’s, because the standard tools really do work and are not hard at all to use.
Really it’s like if people insisted Rust packaging was some horrid hellscape and the moment you asked them what problems they have with Cargo they’re like “Cargo? I don’t use that…”`
I was just using the standard tools (pip) and they broke unexpectedly and took a good amount of effort to fix.
People have built these myriad alternatives because they have gripes with the default tools, like pip installing stuff with root permissions at the system level, or virtualenv causing 8 different identical 2GB copies of Tensorflow to get installed on my system in various random subdirs, etc.
It’s very strange that you choose to defend Python packaging of all things. It’s notoriously bad; people make memes about it. And in the ~8 years I’ve been programming, it’s gotten worse if anything - not even counting the 2->3 transition that people are still paying down the costs of like 15 years later.
I think a part of a problem/perception is that there is quite a lot of blogposts each advocating different approaches. When I was searching some time ago, almost nothing pointed to authoritative sources (at least not the top results where from python.org or pypa.org). When I looked for an inspiration how my colleagues approach building/testing/publishing a python package/app, I was again met with thousands ways of entering the world of hurt — because misinformation spreads easily and is hard to fix.
It’s a Docker container. It is the whole universe as far as packing is concerned. The fact that I have to create a virtual environment inside my Docker container so I can install a Snowflake Python library is ridiculous to me.
It’s a Docker container. It’s an entire isolated operating system, and it turns out a lot of Linux distros use and rely on tools written in Python, which makes messing with the modules available to the default system Python a very bad idea. So you use a venv, which is the standard recommended thing and has been for years.
Heh, this is ironic, no? Docker containers are there to provide packaging, isolation, and distribution of apps. The fact that many use them as a replacement for a VM just shows people don’t follow the best practices. Just the same as with pip…
If one uses docker containers in the right way, having a venv inside of it really feels wrong, because the container should be an unit of isolation.
Like I said, the issue is that Python is both a language people want to write applications in, and a language distros want to write system tooling in. Nobody would be surprised if you told them that replacing key .so files in a system shared location might break things. Why are people surprised if you tell them that replacing key .py files in a system shared location might break things?
So the Docker container is not, in itself, a sufficient unit of isolation for Python code, and never was. Which is why creating a venv is such a strongly recommended practice (and now enforced by some distros).
The fact that distro managed packages for language libraries is a thing at all seems ridiculous to me. It’s a losing game. Some libraries end up specially anointed because they have a maintainer for some distro, some dont, there’s no relation between the annoited ones and the good ones*, and the distro ones are always out of date. It puts languages in a weird place because they want to support their own packaging ecosystem, and also interoperable with a ton of distros, which is a drawn and quartered specification. So like, either distros manage packages and anything else you have to install from source (hope your distro has the right build system and compiler, resolve your deps manually), or languages have environments they can self manage. NPM spearheaded this wacky idea of “why not both” in which if you sudo it, it will futz around in directories that really should be distro managed to fake up a “global install” concept, and python followed. This has since been regarded as a bad idea, and this is the band-aid rip. It never was what it pretended to be, you got lucky if it ever worked for you, I haven’t used snowflake, maybe they figured something out I didn’t see anywhere else.
for a fun time, find the python fuse library that works on freebsd. There is one. It’s not the one in ports.
Edit: I don’t know how to make the asterisk not a bullet :/
I find that the header file problem is one that tup solves incredibly elegantly. It intercepts filesystem calls, and makes any rule depend on all the files that the subprocess accesses. Solves headers in an incredibly generic way, and works without requiring hacks like -MMD.
Not sure if the author is here, but if you are, any plans to support something like that?
It intercepts filesystem calls, and makes any rule depend on all the files that the subprocess accesses. Solves headers in an incredibly generic way, and works without requiring hacks like -MMD.
So the “proper” way is to intercept the filesystem calls in a non-portable manner and depend on anything the program opens without regard for whether it affects the output or not (like, say, translations of messages for diagnostics). While explicitly asking the preprocessor for an accurate list of headers that it reads is a hack?
The problem with the second option is that it isn’t portable between languages or even compilers. Sure, both GCC and clang implement it, but there isn’t really a standard output format other than a makefile, which isn’t really ideal if you want to use anything that isn’t make.
It’s an unforunate format, but it’s set in stone by now, and won’t break. It has become a de facto narrow waist with at least 2 emitters:
Clang
GCC
and 2 consumers:
Make itself
Ninja has a very nice and efficient gcc -M parser
Basically it’s an economic fact that this format will persist, and it certainly works. I never liked doing anything with it in GNU make because it composes poorly with other Make features, but in Ninja it’s just fine. I’m sure there are many other non-Make systems that parse it by now too.
That’s a fair point, also didn’t know Ninja supported it but it makes sense. I wonder if other languages support something similar to allow for this kind of thing, though many modern languages just sidestep the issue all together by making the compiler take care of incremental compilation.
Most tools could probably read the -M output format and understand it quite easily. It doesn’t use most of what could show up in a Makefile - it only uses single-line “target: source1 source2” rules with no commands, no variables, etc. I imagine if someone wanted to come up with a universal format, it wouldn’t be far off from what’s already there.
But.. don’t you want to update your program when diagnostic messages are changed? The FUSE mount doesn’t grab eg. library and system locales from outside the project root, so it only affects the resources of the project being built[1]. Heaven forbid you’re bisecting a branch for a change that is, for reasonable or cursed reasons alike, descended from one of those files..
For those interested, I’ve pitched tup and mused about this in a previous comment here.
[1]: Provided you don’t vendor your all dependencies into the repo, which I guess applies to node_modules! Idk off the top of my head if there’s a way to exclude a subdirectory for this specific situation, or whether symlinks would work for controlling the mechanism.
Edit: Oh, it’s u/borisk again! I really appreciated your response last time this came up and hope you’re doin’ great c:
Edit 2: Oh, and you work on a build system! I’ll check it out sometime ^u^
I originally started Knit with the intention of supporting automatic dependency discovery using ptrace. I experimented with this with a tool called xkvt, which uses ptrace to run a list of commands and can generate a Knitfile that expresses the dependencies. However, I think this method is unfortunately more of a hack compared to -MMD because ptrace is non-portable (not well supported/documented on macOS and non-existent on Windows) and has a lot of complexity for tracing multithreaded processes. A Fuse-based approach like the one used by Tup is similar (maybe more reliable), but requires Fuse (a kernel extension), and also has the negative that automatic dependency discovery can sometimes include dependencies that you don’t really want. When I tried to use Tup for a Chisel project I ran into problems because I was invoking the Scala build tool which generated a bunch of temporary files that Tup required to be explicitly listed as a result.
I think if Knit ever has decent support for an automatic dependency approach, it would be via a separate tool or extension rather than directly baked into Knit by default.
I’ve kinda done something like this over the years, only less purposefully. And I thought I’d settled on Nim, but I rage-quit it a few months ago after a particularly egregious “it’s my language and I can put in any footguns I want*” decree by the Benevolent Dictator-For-Life on the forum.
Anyway for now I’ve ended up at Swift. (And yet I keep coding in C++ because Reasons. Sigh.)
I wonder why the Swift version is so slow. Arithmetic is overflow-checked by default, and arrays bounds-checked, but the OP said they turned that off. Maybe using (ref-counted) classes where structs would have sufficed? Or passing arrays around in a way that defeats the COW optimizations?
* BEGIN RANT There’s a newish Nim feature that lets you declare references as non-null, and uses control-flow analysis to prevent you from assigning/passing a null value, at compile time. Awesome! Only, the analysis was failing in lots of my code and complaining a variable might be null, when I had just checked it right above. After I made some reduced test cases and reported the problem, the BDFL told me (brusquely) that the control flow analysis ignores “return” statements. And that this is not a bug and will not be fixed … because you should be assigning to “result” instead of early-returning (praise Wirth!) This despite “return” being a perfectly cromulent part of the language that’s been around for ages. At this point I decided a single BDFL, esp. a cranky German one [I’m from a German family myself, I know the type] is probably a misfeature and I should look elsewhere. END RANT
This must be what he means – but it comes up all the time. There must be a dozen github issues/RFCs/forum threads where it is referenced. Araq is so strongly opinionated about this one that it is surely a “backing” example to his line in the Zen of Nim: If it’s hard for the compiler to reason about then it’s hard for people to and so you should not do it.
While I love Nim, I disagree on this line item. I think people can be bad at reasoning in situations where computers can be good, and this can matter to program notation - like closing parentheses in Lisp vs. indentation where humans are just bad at “base-1/unary” after about 3..4 things (hence the 5th diagonal slash after ||||, as another example). Even adamant Lispers will say “Let your editor indent/do that for you!” - a paraphrase of saying “a program is better at counting” - they just recommend a different program from the compiler. ISTM, early return is a similar case (but more in the technical weeds).
If you want to tinker, this looks to be the Swift source. I reproduced the 1.5 hour running time estimate by commenting out line 364’s call to verbose printing function output_state_choice. Commit history suggests this was left out during the test. Despite some references to the number of cores in code, I found it used just one core, though I don’t know how the other implementations behaved. Memory grows steadily for the at least the first minute or two, so you could be onto something with copy-on-write behavior.
For me, the Nim hits ~590% utilization (if 1core=100%). I boosted NUM_THREADS from 6 to 16 on a 16-core and that util didn’t change. So, making the work partitioning more fine-grained could maybe yield a 10x better time on a 64-core AMD 3990X – depending upon contention, of course. { Famous last words, I know! :-) }
Having and maintaining your own private language seems like a bad idea. And unless you have a LOT of free time and some very good ideas, trying to attract supporters away from an already-niche language seems like a bad idea, too.
I disagree. If one or two central people that maintains an open source project are not easy to cooperate with, then it can be very fruitful over time if someone forks it.
Also, forking a project does not necessarily mean that a single maintainer needs to do all the work. Receiving pull requests does not need to be that time consuming.
In addition to this, some forks can be maintenence/“LTS” projects, they don’t have to keep the same pace of development to be useful. Sometimes a few selected patches going in can mean more to users than a continous stream of features.
You’re welcome to D. The language is awesome. However, such “BDFL” have teratons of focused complaints to answer about so it’s not necessarily a good idea to escalate the problem on the internet instead of being patient and help fix it.
You’re dropping the whole language because of one extremely niche feature 99.9% of developers would never stumble on? You know it’s not the language that looks bad in this story right?
I don’t think you understand the feature he’s complaining about correctly, because it seems to me to be very common, as attested by @cblake’s comment that “it comes up all the time”.
There’s cross-talk here. The specific A) strictNotNil feature supporting early return is (likely) a small niche, and B) early return/structure in general is much bigger (& what I meant by occurring regularly, e.g. here). Lest quick/casual readers be confused, early return/break are absolutely accepted {“cromulent” :-) } parts of Nim - not going away. Araq pushed back on hard work for A) he feels other PLs skip while being overwhelmed to get Nim2 in shape (& did not write a misleading doc in dispute according to git).
@snej’s earlier comment (& that Forum thread) indicate he was ok with incompleteness & improving docs. Dropping Nim was more related to a “cranky single German BDFL” - a feature of a community, not a programming language. (I agree “completely and objectively wrong” was needlessly inflammatory rhetoric, but “put in footguns” is also.) Anyway, like @agent281 I am also sorry to see him leave!
These disagreements are often about division of labor & perspective taking, not “objectivity” (esp. given the current state of human psychology as a science). To connect to my prior example, Lispers also complain about offside rules making macro writing “objectively” harder at a cost of “objectively less readable” syntax. Both compiler writers & language users understandably guard their time, driving much emotion.
I honestly never went back to look at that thread after my last reply. I probably won’t.
Maybe I’ll try Nim again sometime. I turned to Swift on the rebound and wrote a few thousand lines over the holidays (not my first waltz with that language) and largely enjoyed it except for the incredibly awkward APIs for working with unsafe constructs. (I wasn’t bridging to C, just marshaling data for a binary network protocol.) Which is to say I’m not totally happy with Swift. I’m still waiting for the perfect language.
It was the straw that broke the camel’s back. And where did you get the idea that null checking or the “return” statement are extremely niche features? Ok, null checking isn’t on by default in Nim, but it’s basically table stakes in most new languages ever since Hoare’s “billion dollar mistake” went viral.
If a complex condition is always unreachable, I’d prefer to see my compiler or static analysis tool complain about that instead - since, in many cases, having a complex condition that is always unreachable might itself be a bug - rather than treating the condition as correct and the unreachable statement as dead code.
Of course in certain cases - like ‘0 && expr’ or ‘1 || expr’ (including with macros), those kinds of warnings should be suppressed - but in cases like the example in the article, I personally would rather see a warning about the condition itself being suspiciously constantly false.
Ruby does this right “open do..”. Python does it right, “with … do”. Any reasonably wrapped smart resource will catch this at compile time; we used to do this for memory and files in the ’80s. ‘defer’ is just a way to make mistakes.
open is the classic example where not using with will still allow you to do your work. But most libs will do with thing() as actual_object: ... . If you do actual_object = thing() then you’re not getting the object you want but just a manager object.
The way to get around using with is doing actual_object = thing().__enter__(), which is a bit unwieldy.
Ah ok I guess the title says what it literally is but I was hoping it compiled to Go or something. It’s just written in Go and generates code with LLVM. No Go interop at all. Slightly disappointing (only because I’m predicting the next hit language to be an ML on the Go ecosystem a la F# and Scala) but an awesome compiler project nonetheless.
only because I’m predicting the next hit language to be an ML on the Go ecosystem a la F# and Scala
I agree with this, I think Go is a useful target; I may be influenced by the fact that I’m not much of a fan of writing Go as well, but I do think it’ll be a useful target. I spent a fair amount of time getting carML to output Go nicely, and it’ll be something I target in coastML for that reason as well.
I’ve had this idea for quite some time, but not a ton of motivation. I feel like this is so close that I should be jumping up and down to do it… but, time. Probably won’t be me.
char *one = "one";
char *end;
errno = 0; // remember errno?
long i = strtol(one, &end, 10);
if (errno) {
perror("Error parsing integer from string: ");
} else if(*end) {
fprintf(stderr, "Error: invalid input: %s\n", one);
}
For a similar expected behavior?
The one == end is kind of redundant with *end != '\0'.
Might not be perfect, but API-wise this is not as smelly as it is presented, in my opinion. Actually with C limitations it kind of makes sense. Any parsing error due to numbers that cannot get represented in a long are passed into errno, which is a common way to represent errors in stdlib. Using the end pointer seems like a reasonable and very flexible way to let the caller deal with what should be considered an invalid string. In some case, someone might not care about trailing character (so could use end == buf instead of *end).
SLIME, as the expression of roughly 50 years of parallel evolution and refinement of a programming language, developer toolchain, and ecosystem, is absolutely the gold standard. It’s also had 50 years to get there.
Let’s furthermore not forget that homoiconicity and image-based development has meant that your source code, object code, parser, compiler, debugger, and REPL all ship as part of a single mutually-referential whole. That in turn lets you do exactly the kind of whole-program analysis that the RA article bemoans as largely intractable in Rust-land.
C as a language lags by at least 15 years, and Rust is only a smidge over 10 years old. If you want to find something closer to the Lisp/SLIME universe it’s probably in the JS or .NET ecosystems…neither of which is actually fully self-hosting AFAIK, and so you still hit a hard wall of abstraction when you drop below the standard syntax and libraries.
If we’re very lucky, perhaps Rust + $EDITOR[1] circa the ~2040 edition will have achieved something like the smooth integration and workflow that SLIME devotees have been (rightfully) crowing about since at least the late 90s. :)
[1]: I still don’t see VS Code being this One True Editor, but neither Emacs (given the Lisp-ness discussed above) or (N)Vim seem likely to naturally fill the gap either, since they aren’t born of the same zeitgeist and aesthetic as Rust. Perhaps it’s no coincidence that so many folks in the Rust community are focused on text processing and GUI frameworks, and the ideal “Rust environment” is both imminent and inevitable.
Forgive me, I don’t know slime well, but wouldn’t it have the same problem? Assume some macro expands its arguments by reversing the names of symbols. How would slime know to autocomplete the reverse of what is intended?
This is why Rust’s ecosystem is fundamentally and unrecoverabky broken and which is why I’m glad to be able to rely on my package manager and large standard library in C.
Probably the only reason why Rust crates aren’t pwned to oblivion is the fact nobody cares about Rust other than evangelists and those who still have to find out how messy it is.
To provide an alternative, look at Ada, an absolutely beautiful and secure language with many great ideas and concepts.
Probably the only reason why Rust crates aren’t pwned to oblivion is the fact nobody cares about Rust other than evangelists and those who still have to find out how messy it is.
Okay, I don’t really care about Rust, but I doubt this. If I were to hazard a guess, I’d say that the real reason is more likely that executing effective attacks this way is complicated, it’s still a fairly new area, and the Rust community is waking up and mitigating some of the flaws in what is undoubtedly a flawed system.
If nothing else, there is a lot of money to be made successfully exploiting Rust, given the number of high-profile projects now adopting it. Like it or not, Rust got past the ‘only evangelists and idiots’ phase a long time ago, and the teams using it are, with the best will in the world, not going to rewrite their code in Ada. I don’t see the point in writing it off like this.
It’s a little bit easy to just point at the shortcomings of one side but ignore the other. You’re trading a mature ecosystem that allows git commit tagging to something that doesn’t even have a definition of external library except for linking with what ever is currently in your path variables. Which is neither secure nor reproducible.
What you’re suggesting is that we should all go and rely on your package manager not to break or get pwned (which did happened multiple times in debian). This kind of build system has no idea of versions, security updates, revocation and such, it just ships what happens to be the thing that got into your package management, together with all the changes they thought are useful. (And sometimes, they actually make it worse.)
There is much to be said about securing build scripts and macros, but most of the things shown in this article also work with c (make scripts, macros, init functions..). And due to all the shortcomings of C it is even easier to hide something in there. When you use what debian ships for TLS, you hopefully trust the debian people (and transitively the authors) and you trust people building your software to have an up to date version of it. When you pin stuff in rust, you trust the people running crates.io (see debian) and the authors that put stuff up there. But your users (or upstream distro) will have to actively select a bad version when building it, to get the same effect. Pick your poison.
In a perfect world we sandbox any build scripts and macros and use the rust-sec extensions for crates.io to verify the trustworthiness of our dependencies. At that point the only thing that can go wrong is a malicious build, built inside your sandbox, delivered to everyone.
To catch issues like this sooner, I have automated your manual step. Instead of running a manual dry-run, looking through it for unexpected changes, and then running a manual real-run, I have my backup script email me the list of files that changed since the last backup as it does the backups, no manual step required. This lets me peruse the email when I have a free minute or two, and it doesn’t take long since this happens daily and the files that SHOULD have changed are fresh in my mind. And if I notice anything suspicious I can go check things out and restore from a previous version if anything is amiss. Seems like the right balance to me.
That’s a nice idea, thank you. Various cron jobs already email me from my servers, but it didn’t occur to me to try to email myself from the iMac. That would be easy enough and, as you say, a better balance between automated and manual.
C++ is much older than java and I’d be surprised if it wasn’t in the same performance ballpark than Rust for optional. Old age is not an excuse for bad design.
Are you sure it is bad design? Perhaps Java’s design just makes different trade-offs for performance (like consistency or lower mental model overhead or… I’m sure there could be many).
why would the stdlib implementation matters ? what matters is that as a C++ dev I can just do optional<int>, optional<float>, optional<std::string>, optional<whatever_type> while in Java I have to remember that I must use OptionalInt, OptionalLong, OptionalDouble, and Optional<T> for everything else.
? Of course it can, boost::optional did this for e.g. storing optional<T&> in a pointer for years. Sure, it’s a bit of work to specialize std::optional for your NonZero type but there’s nothing technically impossible
Concepts help, but they are quite recent. Also, C++ has objectively quite a few more footguns than the other languages, despite individual preferences.
Java has a huge boat-anchor, the JVM. It’s stuck with a bytecode instruction set, binary format, and runtime model designed in the mid-1990s before the language had even seen serious use. There have been minor changes, like clarifying the memory concurrency model, but AFAIK no serious breaking changes.
This makes some improvements, like stack-based objects, either impossible, or achievable only with heroic effort by the JIT and only in limited circumstances. (See also: the performance limitations of Java generics due to the requirement of type-erasure.)
User beware. The hand-rolled recopies showcased here are better than your average hand-rolled recopies I see on many projects, but they still leave a lot to be desired. For example mkdir dir; cp foo dir is not nearly as robust as install -D -t dir foo. But that’s trivial compared to the bash specific syntax in an sh file that should be plain POSIX, or the fact that the simulated argument handling doesn’t work quite the way as the real ./configure script generated by autotools does.
As ugly as it can be sometimes, using autotools (autoconf, automake) is far more robust than hand-rolled scripts will ever be precisely because it has such a long history and wide usage all the edge cases not considered by any one person are already taken care of. And in the end the amount of code you actually have to write and maintain would be less for the examples given.
That is almost certainly not autotools fault, but its so confusing to get started using that it doesn’t always get setup right, and that can certainly make it break for users.
Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours. It probably leaves distro packages writing their own build routines and different OS layouts hacking around the over-simplified model shown in 12 lines. And I say this as someone who wrote THOUSANDS of lines of hand-brew makefiles before finally turning to autotools and finding it solved so many of the edge cases I was working around it really was useful.
That is almost certainly not autotools fault, but its so confusing to get started using that it doesn’t always get setup right, and that can certainly make it break for users.
I’d argue that if it is confusing/hard to use, it is the fault of the tool.
How important is it that a given piece of software be buildable on the maximum number of architectures?
Just as important as it is to have access to more systems that x86_64, and run more operating systems than Windows (or, whatever, Aarch64 and macOS). Portability isn’t just about being able to build libpng for OpenVMS in 2021, it’s also about being able to build whatever is going to be important, under whatever operating system is going to be important, in 2025.
You could presumably drop support for the former without causing much of a fuss, but autotools & friends allow us to support any number of architectures, whether maximum or not. If we start to package everything with 12-line makefiles we might as well all buy x86_64 laptops running whatever Linux distro op is running and give it up. Then in 2031 we can all gather here and whine how the next best operating system supports a direct-to-brain interface but is held back by the fact that most software only works with Ubuntu and Docker (or, realistically, Windows and macOS…)
Just as important as it is to have access to more systems that x86_64, and run more operating systems than Windows (or, whatever, Aarch64 and macOS).
I don’t think I understand your point. I agree that it’s important to support more than a single platform. But my claim is that there’s diminishing returns beyond a certain point. Do you dispute that? If not, where is that point? I guess I’m claiming it’s somewhere around the top 5-10 in terms of userbase. Do you think it’s more than that?
My point is two-fold (sorry, it may have been useful to be a little more clear).
First, “the maximum number of architectures” is just another way of saying “any architectures one needs”. Any system, no matter how useful, including Linux, started by not being even close to the top 5-10 in terms of userbase. If you restrict your build system’s capability to supporting only the leading two or three platforms of the day (realistically, a short Makefile will at most get you Linux, BSDs, and maybe Cygwin, mingw & friends need quite a few hacks for non-trivial programs – 5-10 is already way more than you can realistically manage for non-trivial software), you’re dooming it to obsolescence as soon as the next different enough thing hits the market.
I bet that most of the programs that use autotools don’t actually run (at all, let alone well) on all platforms that autotools can build them on. That’s only as big a problem as the users of said software make it to be though, presumably nobody is losing too much sleep over not being able to run qemu on 4.4 BSD. However, if and when you need, you’re at least gonna be able to build it and you can go right ahead and do the substantial porting work you need. Clunky though they may be, programs built using autotools are gonna compile on Fuchsia 3.1 for AR Workgroups or whatever pretty much out of the box (barring any platform-specific things that need code, not build system incantations, of course). Beautiful, minimalistic, hand-rolled, organic Makefiles are gonna be a curse as soon as Ubuntu 20.04 LTS will be a distant memory instead of something that was pretty cool five years ago,
It’s not like we haven’t seen this already, autotools became a thing because, terrible though it may have been even then (like, m4, seriously?) it was still better than managing Makefiles for even two or three commercial Unices (which meant way more than just the OS back then – also different libcs, different C compilers and so on). It didn’t come up in a vacuum, either, it was based on (or rather tried to improve) a bunch of prior art (Metaconfig, imake).
Second, and strictly regarding the number of platforms: in my experience, while the tooling required to build on 1-2 platforms vs. 5-10 platforms is massively different, past that point, the complexity of the tooling itself is pretty much the same, whether you support 5-10 platforms or 50-100. The number of platform-specific quirks obviously increases with the number of platforms, but extending a tool that already supports 10 platforms so as to allow it to support 100 is largely an exercise in implementing platform-specific hacks, whereas extending the tooling (“12-line makefile”) to support 10 platforms instead of 2 basically amounts to writing a new build system.
So while it takes a lot of effort to go from supporting 1 platforms to 5-10, there’s not that much effort in ensuring your software can be built on more than that. If you’ve done that, you might as well go all the way and use a system that has the traction and popularity required to work everywhere for the foreseeable future. autotools isn’t the only one. CMake is another, for example. It’s terrible but lots of popular software is built with it. Anyone who wants their platform to be relevant will have to ensure CMake-built software runs on it, otherwise they’re gonna be stuck without some leading DBMSs, web browsers and so on.
I think I understand your point here. But I think the difference between our positions is that you’re judging autotools by its asserted capabilities, and I’m judging it by it’s practical drawbacks. In my experience autotools fails more often than it succeeds, and it delivers net-negative value to projects that use it. I’m sure your experience is different!
If you restrict your build system’s capability to supporting only the leading two or three platforms of the day (realistically, a short Makefile will at most get you Linux, BSDs, and maybe Cygwin, mingw & friends need quite a few hacks for non-trivial programs – 5-10 is already way more than you can realistically manage for non-trivial software), you’re dooming it to obsolescence as soon as the next different enough thing hits the market.
I don’t think I buy this. Software is a living thing, and if my project is important enough to stand the test of time, it will grow build capabilities for new architectures as they become en vogue.
I think so. While it’s definitely far from my favourite build system, I can’t remember the last time I’ve seen it fail. However, it’s so widespread that it’s very easy to be lucky and only encounter it in scenarios where it works fine.
Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours.
When someone else’s 12 line makefile doesn’t work, it’s clear how to fix it. The same isn’t true for autotools, cmake, or many of the other more complicated alternatives.
tbh I don’t actually agree this. make unfortunately is quite difficult to debug without turning to external tools. Honestly, I find both cmake and autotools a lot easier to debug than just a raw Makefile, since they have debugging facilities.
That said, I don’t turn to cmake or autotools until I hit the level of complexity or reach where that matters, and most of the time just use simple Makefiles.
If it is really 12 lines, it is probably less annoying for distros to maintain their own version than to maintain a command line of ./configure .... If something breaks, it’s much easier to fix a Makefile with 12 lines than the whole auto* setup.
This is worthy of a blog post, but while technically you CAN just go gcc -shared libhober.so libhober.c and call it a day, there’s a lot of subtleties. Specifically:
Your binary will export everything by default because symbol visibility sucks.
Your binary is one architecture only - AIX actually has fat libraries, but not binaries. It works by making libraries .a archives, and you can import based on member.
To circle back a point, you can also put a file in the archive (shr.imp/shr_64.imp - it is very tempting to call them shrimp files) indicating what should be imported, and its name to link against (we’ll come back to that later.)
Want versioning? GNU fucked it up by coming up with two schemas: libhober.a(libhober.so.69) and libhober.so.69 (with an implicit shr.o/shr_64.o depending on arch). The former scheme is insane to manage via a package manager, so everyone sane went for the latter.
Problem: if you do cc -lhober, it’ll link against libhober.so, not libhober.so.69. This works, but is subtly wrong (don’t have the devel package installed, what if soname bumps for a reason?), but it’s easy not to catch because it works in development.
To work around that, your shrimp file can declare the real name of the library to link against.
This kinda makes sense but is unnecessarily brittle. It does make for a great smoke test of how committed to portability you are though. libtool is like violence, it’s the cause and solution to all of life’s problems.
Your binary is one architecture only - AIX actually has fat libraries, but not binaries. It works by making libraries .a archives, and you can import based on member.
I mentioned using “-G” and “ar” to specifically create AIX-like shared libraries, which doesn’t suffer from this issue.
Some of the others may be issues if you need some of those features, but I don’t see how that really precludes you from building a “proper shared library” that doesn’t use those features with just the compiler and archive tools and a really simple makefile.
I think most people using the really simple makefiles probably aren’t using -G and ar to make a shared library (do you think Makefile curators give a shit about AIX?), and now you have to curate your shrimp file for exports (careful, it’s REALLY easy to accidentally end up exporting strcpy if you do all, because strcpy is only a static object with AIX libc :) ). Hell, stuff like CMake barely does it.
The original position was that shared libraries on AIX are hard or impossible without libtool - but my position is, a simple Makefile can handle that just fine (if the author wants to support AIX explicitly, of course). Now, using libtool may give you different platforms like AIX automatically, but that doesn’t support the claim that a simple Makefile couldn’t build “proper shared libraries” on AIX, if desired.
strcpy is only a static object with AIX libc
Are you sure?
$ dump -Tv test.a
test.a[test.o]:
***Loader Section***
***Loader Symbol Table Information***
[Index] Value Scn IMEX Sclass Type IMPid Name
[0] 0x0000fc00 undef IMP XO EXTref libc.a(shr.o) ___strcpy
[1] 0x20000244 .data EXP DS SECdef [noIMid] f
Perhaps it’s changed in 7.2; I remember having plenty of versions like strcpy end up exported in shared libraries naively exporting all, instead of a curated export list or using nm on objects to generate one.
Edit: in some machines, the miniz library does not compile with -Wsign-conversion: you can get it to work by removing that option from CFLAGS in Makefile. This affects only cedro-new: both cedro and cedrocc compile before that error.
Why does the deferred resource release not use __attribute__((cleanup))? It can generate code that is correct in the presence of exceptions, whereas the output code here still leaks resources in the presence of forced stack unwinding. Is it just that you’re doing token substitution (__attribute__((cleanup)) takes a pointer to the cleanup object, so you can have a single cedro_cleanup function that takes a pointer to a structure that contains explicit captures and a function to invoke). The choice of auto for this also means that it cannot be used in headers that need to interoperate with C++ - was that an explicit design choice?
Similarly on the lowering, binary includes are a much-missed feature in C/C++, but the way that you’ve lowered them is absolutely the worst case for compiler performance. The compiler needs to create a literal constant object for every byte and an array object wrapping them. If you lower them as C string literals with escapes, you will generate code that compiles much faster. For example, the cedro-32x32.png example lowered as "\x89\x50\x4E\x47\0D\x0A\x1A..." will be faster and use less memory in the C compiler. I’m not sure I understand this comment though:
The file name is relative to the current directory, not the C file, because usually binary files do not go next to the source code.
What is ‘the current directory’? The directory from which the tool is invoked? If so, that makes using your tool from any build-system generator annoying because they tend to prefer to expand paths provided to tool invocations to absolute paths to avoid any reliance on the current working directory. I don’t actually agree with the assertion here. On every codebase I’ve worked where I’ve wanted to embed binaries into the final build product, those binaries have been part of the source tree so that they can be versioned along with the source.
C has setjmp(3) and friends as a very raw, low-level exception mechanism. It’s basically the same underpinnings, with a much less developer-friendly interface. Still, real C code does make use of it!
And __attribute__((cleanup(..))) doesn’t work with longjmp. Not even C++ destructors run if you longjmp out of a scope. (Both destructors and __attribute__((cleanup(..))) run when unrolling due to an exception though.)
C’s longjmp doesn’t do any stack unrolling, it essentially just sets the instruction pointer and stack pointer in a way which breaks lots of stuff. That’s not to say it’s not used though; I’ve encountered it myself with the likes of libjpeg.
C does not have exceptions, but C often lives in a world where exceptions can be thrown. If your C code invokes callbacks from non-C code, it often has to handle exceptions being thrown through C stack frames, even if the C code itself doesn’t handle them. Writing exception-safe code is one of the main motivations for __attribute__((cleanup)) existing.
It’s certainly not defined behaviour, since the C++ standard doesn’t (naturally enough) concern itself with specifying what happens when an exception passes into code written in another language.
In practice, functions in C may not have unwind information and an exception that propagates into them will be treated the same as an exception that (would) propagate out of a noexcept-attributed C++ function. (Off the top of my head, I think the result is that std::terminate() is called).
However, it is possible to compile C with unwind info (eg gcc has -fexceptions to enable this), and in that case implementations will allow propagating an exception through the C functions. At some point the exception still needs to be caught, which you can only do in C++ (or at least non-C) code, so this is only something that’s needed when you have a call pattern like:
C++: main() {
-> foo() [call into C code]
C: foo() {
-> bar() [call into C++ code]
C++: bar() {
(throws exception)
Well-defined in what context? In the context of the C specifications, any interaction with code not written in C is not well defined. In C++, functions exposed as extern "C" impose some constraints on the ABI, but no strong definitions. All (well, almost all) functions that are part of the C standard are also part of the C++ standard. This includes things like qsort, and qsort in C++ explicitly permits the callback function to throw exceptions.
More relevant, perhaps, are platform ABI standards. On Windows, throwing SEH and C++ exceptions through any language’s stack frames is well-defined behaviour and the C compiler provides explicit support for them via non-standard extensions.
Most *NIX platforms use the unwind standard from the Itanium C++, which defines a Base ABI that is language agnostic and layers other things on top. This defines DWARF metadata for each frame that tells the compiler how to unwind. You can write these by hand in assembly with the .cfi_ family of directives. If a frame doesn’t contain any code that needs to run during stack unwind, this information just describes how to restore the stack pointer, what the return address is, and where any callee-save registers that the function used were stashed. The unwinder can then walk the stack and restore the previous frame’s expected register set and recurse. With GCC and Clang, __attribute__((cleanup)) emits the same metadata that a C++ destructor would in these tables, so the code in the cleanup function is run during stack unwinding. You can use this to do things like release locks and deallocate memory in C if a callback function that you invoke from any language that has exceptions throws an exception through your stack frame. Note that this doesn’t let you catch the exception. There’s no major reason why you shouldn’t be allowed (from C) to block exception propagation, though if you need this then invoking a callback via a tiny C++ shim that does try { callback() } catch (...) {} will prevent any exceptions (even non-C++ ones that use the same base ABI) from propagating into the C code.
Why does the deferred resource release not use __attribute__((cleanup))?
Because that is compiler-dependant, at least for now. I want something that would work where a newer compiler is not available.
Also, the current mechanism in cedro allows code blocks with or without conditionals, which are more flexible than the single-argument function required by __attribute__((cleanup)), unless I misunderstand that feature. I’ve actually never used variable attributes in my own programs, only read about it, so I might be missing something.
The choice of auto for this also means that it cannot be used in headers that need to interoperate with C++ - was that an explicit design choice?
It wasn’t, the reason was to avoid adding more keywords that would be either prone to collisions, or cumbersome to type. For use with C++, you could write the output of cedro to an intermediate file which would be standard C. I’ll have to think about it in more detail to see how much of a problem it is in practice.
the way that you’ve lowered them is absolutely the worst case for compiler performance. The compiler needs to create a literal constant object for every byte and an array object wrapping them. If you lower them as C string literals with escapes, you will generate code that compiles much faster. For example, the cedro-32x32.png example lowered as “\x89\x50\x4E\x47\0D\x0A\x1A…” will be faster and use less memory in the C compiler.
I did not realize that, you are right of course!
I know there are limits to the size of string literals, but maybe that does not apply if you split them. I’ll have to check that out.
EDIT: I’ve just found out that bin2c (which I knew existed but haven’t used) does work in the way you describe, with strings instead of byte arrays: https://github.com/adobe/bin2c#comparison-to-other-tools
It does mention the string literal size limit. I suspect you know, but for others reading this: the C standard defines some sizes that all compilers must support as a minimum, and one of them is the string literal maximum size. Compilers are free to allow bigger tokens when parsing.
I’m concerned that it would be a problem, because as I hinted above my use case includes compiling on old platforms with outdated C compilers (sometimes for fun, others because my customers need that) so it is important that cedro does not fail any more than strictly necessary when running on unusual machines.
Thinking about it, I could use strings when under the length limit, but those would be the cases where the performance difference would be small.
I’ll keep things like this for now, but thanks to you I’ll take these aspects into account.
EDIT END.
What is ‘the current directory’? The directory from which the tool is invoked? If so, that makes using your tool from any build-system generator annoying because they tend to prefer to expand paths provided to tool invocations to absolute paths to avoid any reliance on the current working directory. I don’t actually agree with the assertion here. On every codebase I’ve worked where I’ve wanted to embed binaries into the final build product, those binaries have been part of the source tree so that they can be versioned along with the source.
I see, that’s again something I’ll have to consider more carefully. I keep the binaries separated from the source code, but it would make sense to put things like vertex/fragment shaders next to the C source.
Also, the current mechanism in cedro allows code blocks with or without conditionals, which are more flexible than the single-argument function required by __attribute__((cleanup)), unless I misunderstand that feature. I’ve actually never used variable attributes in my own programs, only read about it, so I might be missing something.
The attribute takes a function, the function takes a pointer. That’s sufficient to implement a closure. For example, you could transform:
int a;
int b;
auto a += b;
Into something like this:
// These can be in a header somewhere
struct cedro_cleanup_capture
{
void(*destructor)(struct cedro_cleaup_capture*);
void *captures[0];
};
void cedro_cleanup(void **p)
{
struct cedro_cleanup_capture **c = (struct cedro_cleanup_capture **c)p;
(*c)->destructor(*c);
}
// Generated at the top-level scope by Cedro
static void __cedro_destructor_1(struct cedro_cleanup_capture *c)
{
// Expanded from a += b
*((int*)c->captures[0]) += *((int*)c->captures[1]);
}
...
int a;
int b;
// Generated at the site of the `auto` bit by cedro:
__attribute__((cleanup(cedro_cleanup)))
struct { void (*destructor)(struct cedro_cleanup_capture); void *ptrs[2] } = { __cedro_destructor_1, {&a, &b} };
Now you’ve got your arbitrary blocks in the cleanups. If your compiler supports the Apple blocks extension then this can be much simpler because the compiler can do this transform already.
It wasn’t, the reason was to avoid adding more keywords that would be either prone to collisions, or cumbersome to type. For use with C++, you could write the output of cedro to an intermediate file which would be standard C. I’ll have to think about it in more detail to see how much of a problem it is in practice.
The best way of doing this is to follow the example of Objective-C and use a character that isn’t allowed in the source language to guard your new keywords. A future version of C may reclaim auto in the same way that C++11 did, and some existing C code uses it already, so there’s a compatibility issue here. If you used $auto then it would not conflict with any future keyword or identifier.
What is ‘the current directory’? The directory from which the tool is invoked? If so, that makes using your tool from any build-system generator annoying because they tend to prefer to expand paths provided to tool invocations to absolute paths to avoid any reliance on the current working directory. I don’t actually agree with the assertion here.
After thinking about it, my conclusion is that you are right, so I have changed the program: now the binary file is loaded relative to the including C source file.
Thanks! What are you currently using this for? The place I would imagine it being most useful is for tiny embedded systems that have a C compiler but no C++ compiler. Firmware blobs that want embedding in the final binary are pretty common there.
Well, today I’m continuing work on source code manipulation tools using tree-sitter, which is a parser generator that outputs C parsers. I started with the Rust wrapper but some of the machines where I would like to run it do not have a Rust compiler, some because of the OS, others because of the CPU ISA.
What I’m doing is exploring how much I can cut out dependencies and remain productive. Dependency hell is manageable for a full-time job, but for anything else which I revisit only occasionally it is not acceptable to get derailed by something that used to work, but they changed it and now it does not anymore, and you can not get back to a previous version because of a tangle of up-/downstream dependencies.
The use case of resource-limited machines like microcontrollers and retrocomputers is also a goal: I hope to resume work in that respect soon; like many people, I have a bunch of such machines lying around waiting for me to find some time for them. The intention is that a simpler build chain should make that easier to do as a spare-time job.
And then, binary includes are very useful to cut down on dependencies even on modern machines: one example is for simple GUI applications, where using nanovg and embedding the fonts and images I can get an executable that does not require installation, and depends only on glibc and the various libGL*, libglfw libraries which works well in practice for me. I find this much easier to keep portable than using big complex GUI frameworks, which I admit provide lots of difficult-to-implement features: for some programs though, I find that my choice is not either minimal dependencies with spartan features or more dependencies with complete features, but either minimal dependencies or a non-compiling/non-running program.
For pretty much anything I was using C for 10 years ago (including a Cortex-M0 with 8 KiB of RAM) I’m using C++. C++17 is available on any system with GCC or Clang support. The C++ standard library is sufficiently ubiquitous that it counts as a dependency in the same way that the C standard library does: it’s there on anything except tiny embedded things. It can easily consume C APIs and with modern C++ there are a lot of useful things for memory management and so on.
I do see your point, and I’ve used C++ for decades and expect to keep using it in the future: the improvements in the last years after the stagnation period have made it much more comfortable to use.
Switching to using Lua for Neovim seems like it’ll be really powerful. I’m just a little scared to make that final jump, because there’s way to really go back to Vim after doing this. I’ve already leveraged a bunch of Neovim-only stuff, but this is a lot more extreme.
The Lua support in Neovim is one of the things making me eager to look at it. Lua is a fantastic little language and there’s no reason to be stuck with something like vimscript in 2021.
I looked a bit at lua, and a lot of stuff seemed much more verbose/awkward to me.
I kinda of like VimScript 😅 There are a few annoying parts (line continuations, ugh, and I still forget that stupid a: sometimes), but overall I find it works fairly well, especially since lambdas (already some time ago) and the method syntax (also some time ago, but never ported to Neovim) were added. It’s a fairly small language that fits its purpose reasonably well.
Neovim is claiming that lua is much faster than VimScript. I’ve seen VimScript chug on files with just a few hundred lines, so I’m hoping I can convert my heavier functions to lua and get a speed boost.
I can’t say I ever ran in to issues with this myself; most of the times I ran in to performance issues it’s because you do something like :%s/<something rather complex on a large file>/, but I’m not sure if Lua will really help with that? From what I can tell, most of the performance issues is when people implement stuff like fuzzy matching in VimScript and similar computation-heavy stuff.
The new VimScript9 is quite a bit faster (but still slower than LuaJIT), but I haven’t really looked at it.
if you keep your dotfiles in source control (which I recommend if you aren’t already), you have nothing to lose. If you make the switch and decide later that it’s not for you, just revert back to the previous version of your config that uses vimscript.
I’m not sure what you mean. Sure, you lose some further custom lua config (until you port it to vimscript), but you won’t end up worse off than you were before you switched if you have your config in source control.
I have recently switched not only to Lua, but to Fennel as well and I am pretty happy with result. Now I need to port rest of my config from VimL to Fennel.
Slight, but nothing annoying me. In the past I was trying to cut load time as much as possible, but now I stopped giving a damn. I haven’t tried precompilation though.
As soon as Evil will learn into all my favourite quirks of Vim. I have tested Emacs in the past and there was always some small nitpick that was missing and was irritating me. Now I get the best of two worlds.
Is there a reason that’s a problem? Neovim is a fork of the vim codebase and migrating to it doesn’t seem all that different from upgrading vim and then using plugins that require the new version as their minimum version. Neovim is basically just vim that is making faster progress on cool new features than mainline vim is.
The big gotcha I hit with Neovim was that it uses a different format for undo files. I have this in my vimrc:
set undodir=${XDG_CACHE_HOME}/vim/undo//
set dir=${XDG_CACHE_HOME}/vim/swap//
set backupdir=${XDG_CACHE_HOME}/vim/backup//
set undofile
set undolevels=1000
set undoreload=10000
This avoids cluttering my working directories with undo / backup / swap files and gives me effectively unlimited persistent undo, my favourite feature of vim. If I run neovim on a file that I’ve edited with vim, it complains. Similarly, if I edit with nvim and then vim, vim can’t read the undo file. I wish they’d change the name of files if they were going to have things for the same purpose as vim but with incompatible formats.
That’s what I did, but it’s annoying that I have to.
I was also quite excited by the built-in LSP support, but all of the tutorials I found required incredibly complex configs and didn’t explain how to configure the LSP integration for something like clangd for C/C++.
What are some neovim features that you’ve really liked? I generally see the same things trotted out by fans, like treesitter and in-build LSP, but these aren’t things I’ve felt like I was missing in Vim (I do use LSP, plugins work well for this). I’d love to hear what benefits you have found, as an experienced Vim user?
It’s a bit of a moving target, because a lot of neovim innovations are eventually implemented in vim. That’s what happened with floating windows, async jobs, and terminal emulation. So I guess the feature I really like is being a couple years ahead of everyone else :)
There’s also a few smaller things: I really like inccommand, which I don’t think has been ported over yet. And I just found out I can use the lua bindings to do this:
local nav_keys = {'h', 'j', 'k', 'l'}
for _, key in ipairs(nav_keys) do
local c_key = '<c-'..key..'>'
vim.api.nvim_set_keymap('n', c_key, '<c-w>'..key, opts)
vim.api.nvim_set_keymap('t', c_key, [[<c-\><c-n><c-w>]]..key, opts)
end
I expect that as I get more comfortable with lua and the neovim API I’ll have more say. I write a lot of custom functions and vimL is one of the worst languages I’ve ever used.
EDIT ooh something I just realized I can do:
local set_all = function(list)
for _, item in pairs(list) do
vim.o[item] = true
end
end
set_all {"termguicolors", "hidden", "splitright"}
set_all {"number", "relativenumber"}
No, no inccommand in Vim still. I was quite keen on this for a while until I discovered traces.vim which is a lot more powerful.
Yeah vimL has its quirks. The actual examples you’ve shown would look pretty similar in vimscript though … although they would use :execute, e.g. exe 'nnoremap' c_key '<c-w>..key. I’ve come to quite like vimL but I realise that puts me pretty firmly in the minority. I hate the line continuations but those are going away in vim9script.
It does look like configs will diverge now, to the lua neovim scripts and the Vim vim9script scripts. Which I suppose is good, trying to support 2 diverging editors with a single code-base is annoying, as each add different features.
for k in ['h', 'j', 'k', 'l']
exe printf('nnoremap <C-%s> <C-w>%s', k, k)
exe printf('tnoremap <C-%s> <Cmd>wincmd %s<CR>', k, k)
endfor
For a somewhat creative use of this you can see this ;-)
I guess the issue is that a lot of people just can’t be bothered to invest in learning VimScript, which is fair enough. But pretty much anything you can do in Lua you can do in VimScript, and vice versa.
I guess the issue is that a lot of people just can’t be bothered to invest in learning VimScript, which is fair enough. But pretty much anything you can do in Lua you can do in VimScript, and vice versa.
Also, there’s more third party Lua libraries. A while back I needed to fill an autocomplete based on information in a YAML file, but couldn’t figure out how to parse YAML in VimScript. I ended up writing a python script to first convert the file to JSON, then use json_decode. It worked in that case but would have been nicer to just import a library and read the YAML directly.
I think this - the availability of the Lua library ecosystem - is probably a “bigger deal”, looking forward, than the performance/tooling/ergonomics benefits Lua brings over VimScript. I haven’t seen too much adoption of Luarocks packages in plugins yet (though there are some examples, e.g. https://github.com/camspiers/snap), but I expect this to change now that 0.5 is stable.
I will also say that, while anything* you can do in Lua you can do in VimScript, I personally find it much easier to do most things in Lua - partially due to familiarity, but also due to the simplicity of the language and the relative lack of footguns, compared to VimScript.
*In Neovim specifically, Lua provides access to luv, which I’m not sure you can access from VimScript. In my experience, having the libuv event loop and utility functions readily at hand makes it possible to write functionality which would be at least substantially more painful without these tools.
I haven’t seen too much adoption of Luarocks packages in plugins yet, but I expect this to change now that 0.5 is stable.
Why do you expect this to change with 0.5? It’s been possible to write Lua plugins since 0.3, with luv becoming available in 0.4. I don’t see anything (besides the hype, maybe? :^) ) in 0.5 that would change the attitude of plugin authors towards Luarocks.
It’s true that we’ve had the ability to write Lua plugins for a while now, but there’s been a huge increase in the use of Lua during the later part of the 0.5 development cycle (the past year or so). This, to me, indicates that we may have passed an inflection point in the ease of using Lua for plugin development that will contribute to more people writing Lua plugins, and (potentially) more people using functionality from Luarocks libraries for those plugins (though I admit this also assumes that people need and know about the libraries available through Luarocks).
The hype is another factor, at least at first. And - noting that this runs dangerously close to self-promotion - I think that tools like packer.nvim‘s Luarocks integration (which you can use even if you don’t use packer for plugin management) significantly lower the complexity cost of using Luarocks packages, while also requiring features from 0.5.
I see. In my opinion there’s a fundamental issue with the with dependencies where as a plugin author you don’t want to rely on external dependencies (otherwise you’re inevitably going to end up with people opening issues after misconfiguring the dependencies for your plugin…) but vendoring is fairly unattractive too (messes up version control history, you have to manually update everything…). This issue was already there with Vimscript and I don’t think it can be solved by anything else than Vim/Neovim implementing a full-featured plugin manager (which I don’t think it should do).
Yeah, that’s definitely true too. I think this is easier with Luarocks because it already supports some basic versioning specification, so you can control what versions of dependencies your users get without needing to vendor. I definitely agree that there shouldn’t be a full-featured plugin manager baked into Vim/Neovim, though - people’s preferences and needs around plugin management are too disparate for there to be a good “one size fits all” option, in my opinion.
I think this is easier with Luarocks because it already supports some basic versioning specification
My point was that the package manager that downloads from luarocks itself becomes a dependency, which you want to avoid. So luarocks doesn’t really make things easier there :)
Or Ruby, Racket, Lua, Tcl, Perl, or even Visual Basic. It’s nice that Vim can interface with all that, but the big problem with that is that a lot of people don’t have the scripting support, and if you want things to work on Linux (all distros), macOS, and Windows is nigh-impossible. In reality It’s all pretty useless if as can never use it and have it work for everyone (the reason I didn’t try :lua is because my Vim isn’t compiled with it, and also, because I don’t really know Lua).
Generally Vim tries to minimize required dependencies, which isn’t a bad thing. The Lua interpreter is pretty small so it could be integrated like libvterm or xdiff instead of linking toliblua. But I guess it’s too late for that now :-/
I think under-appreciated aspect is that Rust not only prevents safety issues from being exploitable, but in many cases prevents them from happening in the first place.
Contrast this with C where safety efforts are all focused on crashing as fast as possible. Things are nullable, but you get segfaults. Buffers are unsafe, but you have crashes on guard pages and canaries. Memory management is ad-hoc, but the allocator may detect an invalid free and abort.
Rust in some places ensures safety by panicking too, but it also has plenty of tools that guarantee safety by construction, like move semantics, borrowing, iterators, match on options and results. So the bar is higher than merely “not exploitable because it crashes”. It’s not exploitable, because it works correctly.
Interestingly, “crash early, crash often” is actually AFAIU an approach that can also lead to really robust software when properly embraced, as in Erlang. Personally I’m not convinced C is a particularly exemplary follower of this philosophy, at least as seen in the wild. With that said, I sometimes wonder if this isn’t something of a blind spot for Rust - i.e. stuff like cosmic rays (which I read start becoming noticeable at some scales), or processor bugs and attacks based on them. Maybe some of those risks can be mitigated with stuff like ECC memory? I’m certainly not an expert in this area FWIW, far from it. Just when thinking, I sometimes wonder if the Erlang philosophy is not the winner here in what it can achieve in the biggest horizon of robustness/resiliency (?)
Rust has the “crash early” philosophy too (asserts, indexing panic, overflows can panic), but it just doesn’t need it as often at run time. You wouldn’t want to crash just for the sake of crashing :)
Rust has all of C’s protections too. They’re added by LLVM, OS, and hardware.
Erlang’s “Let It Crash” philosophy is more than just ‘die a lot’ – the other part is that, recognizing that software can fail for a wide variety of reasons but that those reasons are frequently sporadic edge cases, you can get huge operational wins by having an intentional and nuanced system of recovery in-platform.
In practice what this means is that in Erlang/OTP, you frequently have tens, thousands, or millions of concurrent processes, as you might in an OS. Many of these processes are executing business logic, etc. But some of them are ‘supervisor’ processes, which detects when one of the processes it is monitoring has died, and then attempts to restart it (perhaps with a maximum number of attempts in a given time period). This is recursive; for complex applications you might have supervisor trees that are several layers deep.
The benefit here is that you start by writing the golden path. And then, sometimes, you just stop. Get illegal JSON input? Explode. Get an HTTP request you didn’t expect? Explode. Frequently, pragmatically, you don’t need to write the code that handles all of the garbage cases if your supervision tree is solid enough and your lightweight processes start ultra fast.
This philosophy has downsides – if the processes are carrying a large amount of important internal state, then sometimes them dying can be inconvenient because resurrection might involve an expensive rebuild step. So you need to pay more than the usual amount of attention to the stateful/stateless boundaries.
Rust at one point had this as a core idea, but abandoned it for, in my opinion, poor reasons. And now lands up grappling with janky concurrency coloring problems, as it unfortunately appears Zig will too.
Rust at one point had this as a core idea, but abandoned it for, in my opinion, poor reasons.
The OTP monitoring system seems really really cool. However rust abandoned green threads for a very good reason: performance. When the goal is to be able to maximize performance, and compete with C++, you can’t exactly ask for heavy runtime with reduction counters and all the bells and whistles of BEAM, useful as they are.
The “coloring problem” caused by async stems from the same root causes: you can’t really have resizable stacks and lightweight threads in a language that is supposed to be able to run without even a heap allocator (“no-std” in rust parlance). That’s better left to higher-level languages like java or Go (or python if they had made less bad choices).
In short: Rust’s removal of green threads was for two reasons: 1) performance (as you mentioned), 2) difficult interoperation with C. Given Rust’s design constraints, including that you shouldn’t pay for what you don’t use, and that interoperation with C is expected to be easy, this approach didn’t work for Rust.
Edit: Replaced link to thread with a link to a collection, as Twitter’s default threading makes it easy to miss the second half.
Crash-early is fine, as long as the process has a supervisor process that promises to take care of restarting it. That’s the part that seems missing in a lot of systems (in C or otherwise.) Often it just seems kind of ad-hoc. Having implemented some IPC in C++ in the past using POSIX APIs, I remember it being nontrivial for the first process to handle the child’s death (but I may be misremembering.) Its certainly not something done for you, the way it is in Erlang.
Yes, the worst part is when something overwrites some other (valid) memory, causing an error when that memory is next accessed (or worse no error at all). Pretty much your only recourse at that point is to whip out hardware watchpoints (which IME tends to be buggy and tank performance).
Oh, imagine if Mozilla, Microsoft, and Google knew they can just fuzz their C code to get rid of all the bugs! These foolish companies have spent so much money and engineering time on their security programs, patching CVE after CVE, of which ~70% were caused by memory unsafety. If only they knew…
With C it’s always “all these vulnerabilities don’t count”, because you should have fuzzed more, you should have got better sanitizers, you should have got better programmers — for 40 years. The real futility is in “just don’t write bugs” approach.
The safer languages, on top of built-in safety guarantees, also support fuzzing and memory sanitizers, so you can perform your usual due diligence there too.
“Battle-tested” is a weak property. It only applies to software that’s unmaintained. As soon as you change the code, even a single line, that code isn’t proven to be reliable any more (remember when Debian broke SSH key generation to silence a compiler warning?).
Being battle-tested is merely an effect of software getting shipped. So ship the new stuff, and it’ll get battle-tested too.
This is cryptographic code we’re talking about. That fuzzing test suite you need in C? You also need it in Rust. And now the only difference between C and Rust is the fact that with C, you need to run your existing test suite under Valgrind and the sanitizers. Push a button, wait for a minute at most, and you’ll get almost all the safety Rust guarantees.
Because unlike most application code, cryptographic code, thanks to being constant time, is stupidly easy to test. And test it you must, considering the stakes. It is one of the few cases where “just don’t write bugs” is both mandatory and possible.
“Battle-tested” is a weak property.
I agree. What you really want is a kickass test suite.
Test suites are not enough. This is proven time and time again. Any class of bugs you can move from “has to be caught by a test or fuzzer” to “can’t occur in the first place” is a huge win. Cryptographic code should desire these properties even more because it is usually used when security and correctness is paramount.
I would love to go even further and be able to prove constant time algorithms or higher level mathematical relationships statically, etc. Perhaps with theorem provers.
The discussion of the tradeoffs should be around language ergonomics and platform support, not around “but c basically has the same guarantees if you squint”. It absolutely does not.
Cryptographic code really is different. Yes, you want correctness, safety and whatnot. More than for pretty much anything else, save real time avionics and lethal medical devices. Thing is, Rust does not help you there.
The discussion of the tradeoffs should be around language ergonomics and platform support, not around “but c basically has the same guarantees if you squint”. It absolutely does not.
Oh but I am discussing tradeoffs there. C has (as of early 2020) substantially better platform support, significantly worse API, and in the specific case of cryptographic code, a negligible safety disadvantage.
Yes, for most domains, the safety advantage of Rust is freaking huge. I’m only talking about cryptographic code, that I have first hand experience with. Yes, you still need to have some kickass test suite, but Rust would need the exact same test suite anyway. Yes, you need to use sanitizers & Valgrind, but that’s often trivial for cryptographic code, since it tends to have zero dependencies to begin with.
I’m not squinting here, I’m just looking at the application domain at hand. In this particular case, C actually provides basically the same guarantees as Rust. I wouldn’t have expected this 4 years ago when I started working on Monocypher (I actually considered Rust, but favoured C’s portability), but now I can confidently take it for granted: libraries that:
do not allocate on the heap;
have code paths independent from the content of their inputs;
have no dependencies;
are only marginally safer when implemented in Rust, provided they have a good test suite. Modern cryptographic code fits that bill. It’s probably the only niche that does.
That said, cryptographic code doesn’t stand in a vacuum. It must be used somehow. C APIs are naturally unsafe, and the rest of the application will likely hugely benefit from Rust’s guarantees. The crypto library must then have a Rust interface. We could write bindings, but it’s a significant hassle. I would never chose C over Rust for new cryptographic code for a Rust project. For old cryptographic code, I wouldn’t mind considering a reputable C library, though I’d likely favour a Rust library instead.
Would you trust a C crypto lib more than HACL*, a lib written in F* (a dependently-typed language) and formally verified for memory safety, correctness, and resistance to side-channel attacks?
I wouldn’t. Monocypher’s comparative advantage are its simplicity, it’s ease of use, its ease of deployment, and its portability (part of its portability comes from its relatively small code size). I did sacrifice confidence a bit to get there.
I respect your opinion but boy I don’t understand it. I would take a formally verified library any day, assuming it is an option. And I don’t get why someone in the field wouldn’t. Maybe some day I will become enlightened.
I don’t think Daniel Bernstein got a lot of flak when he failed to make NaCl and TweetNaCl formally verified. He’s Daniel Freaking Bernstein after all, world renowned cryptographer who successfully invented primitives. Or maybe because this was 2008, and formal verification wasn’t such a thing at the time?
I don’t think Frank Denis got a lot of flak when he failed to make Libsodium formally verified. After all, it started out as a repackaging of NaCl, itself not formally verified, and expanded since. Not long ago, security advocates were still chanting use Libsodium, use Libsodium, use Libsodium. On the supported platforms, it is indeed a very strong alternative, used by high profile projects and audited by one of the most reputable security companies out there.
Even I didn’t get a lot of flak for failing to make Monocypher formally verified, when I started out over 4 years ago. I did get a lot of flack for daring to write a crypto library without first building a reputation for myself (which is backwards: achievements build reputation, not the other way around). Though that was almost exclusively from people who didn’t even look at my work (as evidenced by their criticism).
Now there is no question that a formally verified crypto library maximises confidence. We can’t beat machine checked proofs. But we can approach them. Here’s how I did it:
Monocypher, like TweetNaCl is small. 2K SLOC. I also made sure I adhere to some subjective coding standard instead of packing everything as tightly as I could, to make sure this metric isn’t entirely bogus.
Monocypher has zero dependencies. Not even Libc. The only code I have to test is my own.
Monocypher, like the entire NaCl family, is constant time (with the exception of signature verification). This makes it much easier for the test suite to have “true” 100% coverage (that is, it covers not only all the code, but all the code paths).
Monocypher is dumb. No clever trick of any kind. The only one I tried cost me a critical vulnerability, and I have since reverted to bog standard implementation techniques that I fully understand.
Monocypher is rigorously tested. The test suite compares with standard test vectors (including Wycheproof, which was designed to catch subtle bugs), tests every possible input & output lengths, uses property based testing to asses consistency, and runs all that under all sanitisers under the sun, Valgrind, and even the TIS interpreter (and now TIS-CI).
Monocypher is partially proven correct. I didn’t wait for proof assistants to make sure its more delicate parts were free of errors. I have written a full proof for Poly1305, the source code for 2^255 - 19 arithmetic contains proof of absence of overflow. I also instrumented the elliptic curve code in an alternate branch to check that the invariants of the modular arithmetic code are respected by the elliptic curve code that uses it.
Monocypher underwent a successful third party security audit.
Is that as good as a formal, machine checked proof of correctness of the generated binary code? Certainly not. But it damn sure approaches it, especially if your compiler is formally verified like ccert. So when I say I sacrificed confidence, that’s how little. While I would hesitate to trust it with the nuke launch codes, I would trust it with my payment card information.
Moreover a project using a crypto library must also consider other sources of errors. If the rest of the program isn’t formally verified, we’re kinda back to square one, where the whole thing must be thoroughly tested to make sure we don’t have some vulnerability or error lurking there. For instance, have we made sure that the network packets can’t be adversarially reordered without us noticing? Using authenticated encryption is not enough, we need message numbers or timestamps for this. So unless you formally prove the whole thing, that vaunted confidence will be significantly lowered, and the difference between a formally verified crypto library and a “merely” rigorously tested one is even slimmer.
As for the advantages, well: Monocypher is a single file library. One source file, one header file. One more of each if you really really need compatibility with Ed25519 (by default, Monocypher uses Blake2b instead of SHA-512 for signatures). Putting it in a C/C++ project is trivial: just copy & past the files in. Language bindings are easy to write, and people wrote several without me even needing to ask. The size of the binary is very small, and further increases Monocypher’s portability. Though it wasn’t the initial goal, it turns out Monocypher is very well suited to small 32-bit microcontrollers: doesn’t take much space, and it is fast.
Now I’m not saying a formally verified library can’t also be extremely portable and very easy to use. Strictly speaking, Monocypher itself could be formally verified (assuming the C standard can be at all formalised). But if it turns out the currently formally verified libraries are harder to deploy, or less well documented (Monocypher has exquisite documentation, thanks to a couple contributors), then the time you take learning how to use it is time you don’t take writing or testing your own program. By being so cheap to integrate, Monocypher can also contribute to the overall confidence of the projects that use it. Or at least lower their costs.
That being said, sometimes the cost of using formally verified cryptography is very low. Jason Donenfeld briefly used Monocypher in the Windows Wireguard installer (to check that the fetched binaries were the right ones), then at some point switched to formally verified code in very little time. As much as I’d like to deny it, this was the right move.
If those people come into a forum where I’m discussing the advantages of safe programming languages and formal verification and they argue what you argued, I would respond the same way. This isn’t a personal attack…
And all of the other text in your post, to me, further strengthens the case for safe languages (assuming ergonomics, portability, etc are on par, as we’ve discussed). Having lots of arbitrary application code to secure, having lots of programmers who would otherwise have to be extremely diligent with tooling, etc - it just isn’t going to happen.
I may have been more salty than I have any right to be, sorry about that. Rest assured, I perceived no such attack. The advantages of formally verified code are real, and I reckon I have a vested interest in ignoring them.
Speaking of which, a more recent post taught me that the first thing they wrote in Rust was the ASN.1 parsing. That kinda changes everything: the advantages of a safe language in this case is substantial. Chacha20 is easy to implement safely in C, but recursive parsing code is a whole ‘nother can of worms. I’ve written parsers, and C is never my first choice there. The compatibility problem sucks, but if they need ASN.1, that was probably the right move. (Modulo a slap on the wrist for failing to use semver.)
(assuming ergonomics, portability, etc are on par, as we’ve discussed)
Incidentally, I believe safe languages have the potential to be even more portable than C currently is. Once you have a formal specification and a formally verified reference implementation, you can make guarantees far beyond C could ever provide. And if you find a way to compile to standard C itself (in a way that doesn’t rely on undefined/unspecified/implementation defined behaviour), then you instantly exceeds its portability, though talking to existing C code may still be a hassle.
Having lots of arbitrary application code to secure, having lots of programmers who would otherwise have to be extremely diligent with tooling, etc - it just isn’t going to happen.
I agree, even in the case of Monocypher: while I have personally been very diligent about its implementation, and can confidently claim it is solid, its interface is definitely unsafe. Because C, and because no dependency: it’s not just the buffer overflows, there’s also the lack of an RNG. Users can’t just generate a key pair, they have to provide the random private key themselves, and I can just pray they stumbled upon one of the many warnings in the manual about how they should use the OS’s RNG. Same problem with authenticated encryption: users have to provide the nonce themselves, and the onus is on them to avoid catastrophic nonce reuse.
Because of this, I strongly believe that a low level crypto library such as NaCl, Libsodium or Monocypher is not enough (they’re higher-level than the low-level operations of OpenSSL, but my work on protocols convinced me they’re still fairly low level). At the very least, I strongly encourage language bindings to provide a higher-level API where users no longer have to worry about providing clean random numbers.
I’m also working on higher-level protocols. Key exchange first, then I’ll tackle a file format. Those can still be free of I/O, small, simple, and in C (with the usual extreme required diligence, which I’ve become better at the last few years). The end game, I think, will be a high-level network library. I’ll have to tackle actual I/O this time, so extreme portability is not possible. Pure C is really not looking good with this one. A Rust or Zig implementation (both of which support async), with a C interface, are probably much better. I’m not sure yet, we’ll see.
I don’t think the original author of the article realizes that cryptographic code is different. His arguments might be right for cryptographic code, but wrong for code in general, which was what he intended. Probably a Gettier problem here.
I am aware of many reasons (though not all of course) where crypto code is different from bc. However, what I do know is that crypto code should be small, thoroughly tested, and frozen. My argument in the post was “I did this amount of due diligence for bc, which is not ‘critical’ code, so have the cryptography authors done that much due diligence?”
First, their argument for Rust (and against C) because of memory safety implies that they have not done due diligence in finding and fixing such bugs.
[…]
And with my bc, I did my due diligence with memory safety. I fuzzed my bc and eliminated all of the bugs.
This seems like such a short-sighted, limited view. Software is not bug-free. And ruling out a class of bugs by choice of technology as a measure of improving overall robustness won’t fix everything, but at the very least it’s a trade-off that deserves more thorough analysis than this empty dismissal.
I think he is probably right (after all, he wrote it) when he says rewriting his bc in Rust would make it more buggy. I disagree this is an empty dismissal, since it is backed by his personal experience.
For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.
I think there’s a big difference between programs and libraries with stable requirements and those that evolve here. The bc utility is basically doing the same thing that it did 20 years ago. It has a spec defined by POSIX and a few extensions. There is little need to modify it other than to fix bugs. It occasionally gets new features, but they’re small incremental changes.
Any decision to rewrite a project is a trade off between the benefits from fixing the accumulated technical debt and the cost of doing and validating the rewrite. For something stable with little need of future changes, that trade is easy to see: the cost of the rewrite is high, the benefit is low. In terms of rewriting in a memory-safe language, there’s an additional trade between the cost of a memory safety vulnerability and the cost of the rewrite. The cost of Heartbleed in OpenSSL was phenomenal, significantly higher than the cost of rewriting the crypto library. In the cast of bc, the cost of a memory safety bug is pretty negligible.
Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs. The author talks about fuzzing, address sanitiser, and so on. These are great tools. They’re also completely unnecessary in a memory-safe language because they try to find classes of bugs that you cannot introduce in the first place in a memory-safe language (and they do so probabilistically, never guaranteeing that they’ve found them all).
If you’re starting a new project, then you need a really good reason to start it in C and pay the cost of all of that fuzzing.
Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs.
There’s an implied assumption here that if a language is memory safe, those memory safe bugs will simply go away. In my experience, that is not quite true. Sometimes those memory safety bugs will turn into logic bugs.
Not to pick on Rust here, but in Rust it is very common to put values into an array and use array indices instead of pointers when you have some kind of self-referential data structure that’s impossible to express otherwise using rust’s move semantics. If you simply do such a naive transformation of your C algorithm, your code will be memory safe, but all your bugs, use after free, etc, will still be there. You just lifted them to logical bugs.
Rust has no good abstractions to deal with this problem, there are some attempts but they all have various practical problems.
Other languages like ATS and F* have abstractions to help with this problem directly, as well as other problems of logical soundness.
Right - but in lifting these from memory bugs to logic bugs, you get a runtime panic/abort instead of a jump to a (likely-attacker-controllable) address. That’s a very different kind of impact!
You don’t get a panic if you access the “wrong” array index. The index is still a valid index for the array. Its meaning (allocated slot, free slot, etc), is lost to the type system, though in a more advanced language it need not be. This later leads to data corruption, etc, just like in C.
It leads to a much safer variant of data corruption though. Instead of corrupting arbitrary memory in c or c++ (like a function pointer, vtable, or return address), you are only corrupting a single variable’s value in allocated and valid and aligned memory (like a single int).
You would get a panic in rust for every memory corruption bug that could cause arbitrary code execution, which is what matters.
This later leads to data corruption, etc, just like in C.
Can you expand on this? I had expected the behaviour in Rust to be significantly safer than C here. In C, the data corruption caused by use-after free often allows an attacker to execute arbitrary code.
I totally see your point about logical corruption (including things like exposing critical secrets), but I don’t follow that all the way to “just like C”. How would an array index error be exploited in Rust to execute arbitrary code?
I have once written a bytecode interpreter in C++, for a garbage collected scripting language. I implemented my own two-space garbage collector. For performance reasons, I didn’t use malloc() directly, but instead allocated a big enough byte array to host all my things. If I overflew that array, Valgrind could see it. But if I messed up it’s internal structure, no dice. That heap of mine was full of indices and sizes, and I made many mistakes that caused them to be corrupted, or somehow not quite right. And I had no way to tell.
I solved this by writing my own custom heap analyser, that examined the byte array and tell me what’s in there. If I see all my “allocated” objects in order, all was well. Often, I would see something was amiss, and I could go and fix the bug. Had I written it in Rust instead, I would have had to write the exact same custom heap analyser. Because Rust wouldn’t have prevented me from putting the wrong values inside my array. It’s perfectly “safe” after all, to write gibberish in that array as long as I don’t overflow it.
Now could this particular bug lead to arbitrary code execution? Well, not quite. It would generate wrong results, but it would only execute what my C++/Rust program would normally execute. In this case however, I was implementing a freaking scripting language. The code an attacker could execute wasn’t quite arbitrary, but it came pretty damn close.
The effects of data corruption depend on what the code does with the data. This often means arbitrary code execution, but not always. It’s not a property of C, it’s a property of the code. This doesn’t change when you change the implementation language.
Fundamentally there is no semantic difference between a pointer in a C heap and an array index into a Rust array. In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code. It’s what the code does with the data that leads to exploitation (or not).
Of course Rust has many additional safety advantages compared to C, buffer overflows don’t smash the stack, etc, and using references in Rust if you can is safe. And when using references, there’s a great deal of correlation between Rust’s notion of memory safety and true logic safety. This is good! But many people don’t realise that this safety is predicated on the lack of aliasing. The borrow checker is only a mechanism to enforce this invariant, it’s not an operative abstraction. It’s the lack of aliasing that gets you the safety, not the borrow checker itself. When you give up aliasing, you lose a lot of what Rust can do for you. Virtually everybody understands that if you introduce unsafe pointers, they give up safety, but less people seem to understand that introducing aliasing via otherwise safe mechanism has the same effect. Of course, the program continues to be memory safe in Rust terms, but you lose the strong correlation between memory safety and logic safety that you used to have.
Not that there’s anything wrong with this, mind you, it’s just something people need to be aware of, just as they are already aware of the tradeoffs that they make when using unsafe. It does make a projection for the number of bugs that Rust can prevent in practice more difficult, though.
I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”. Even a rust alias logic bug dies not open up this kind of arbitrary code execution exposure because you can’t alias an int with a function pointer or a vtable or a return address on the stack, like you can in c or c++. You can only alias an int with an int in safe rust, which is an order of magnitude safer and really does eliminate an entire class of vulnerabilities.
I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”.
In the security research world, we usually treat control of the program counter (the aptly named rip on x86-64) as “arbitrary code execution.” You can do a surprising of programming using code that’s already in a process without sending any byte code of your own with return-oriented programming.
But does Rust let you do that here? What does a snippet of Rust code look like that allows attacker-controlled indexing into an array escalate to controlling the program counter?
Surely you agree that “variables changing underfoot” implies “programs flow becomes different from what I expect”. That’s why we use variables, to hold the Turing machine state which influences the next state.
A logical use after free means “variables changing underfoot”. You don’t expect a free array slot’s value (perhaps now reallocated) to change based on some remote code, but it does.
Right, but “program flow becomes different from what I expect, but it still must flow only to instruction sequences that the original program encoded” is much much safer than “program flow can be pointed at arbitrary memory, which might not even contain instructions, or might contain user-supplied data”.
Maybe not directly within the native code of the program itself, but I think (at least part of) 4ad’s point is that that’s not the only level of abstraction that matters (the memory bug vs. logic bug distinction).
As an example, consider a CPU emulator written entirely in safe Rust that indexes into a u8 array to perform its emulated memory accesses. If you compile an unsafe program to whatever ISA you’re emulating and execute it on your emulator, a bad input could still lead to arbitrary code execution – it’s at the next semantic level up and not at the level of your program itself, but how much does that ultimately matter? (It’s not really terribly different than ROP – attacker-controlled inputs determining what parts of your program get executed.)
That’s admittedly a somewhat “extreme” case, but I don’t think the distinction between programs that do fall into that category and those that don’t is terribly clear. Nearly any program can, if you squint a bit, be viewed essentially as a specialized interpreter for the language of its config file (or command-line flags or whatever else).
There’s no distinction here. If your program implements a cpu emulator then your program can execute with no arbitrary code execution at all and still emulate arbitrary code execution on the virtual cpu. If you want the virtual program executing to not possibly execute arbitrary virtual instructions, you need to generate the virtual program’s instructions using a safe language too.
In most cases, though, arbitrary virtual code execution is less dangerous than arbitrary native code execution, though that’s beside the point.
So…we agree? My point was basically that attacker-controlled arbitrary code execution can happen at multiple semantic levels – in the emulator or in the emulated program (in my example), and writing the emulator in a safe language only protects against the former, while the latter can really be just as bad.
Though I realize now my example was poorly chosen, so a hopefully better one: even if both the emulator and the emulated program are written in memory-safe languages, if the emulator has a bug due to an array-index use-after-free that causes it to misbehave and incorrectly change the value of some byte of emulated memory, that destroys the safety guarantees of the emulated program and we’re back in arbitrary-badness-land.
Sure but this is just as meaningful as talking about a cpu hardware bug that might cause a native safe program to run amok. Technically true but not very useful when evaluating the safe programming language
Yeah exactly, this is the part I thought @4ad was arguing was possible. Eg. in C, use-after-free often allows me to make the program start interpreting attacker-provided data as machine code. I thought this is what 4ad was saying was also possible in Rust, but I don’t think that’s what they are claiming now.
To me, that’s a big difference. Restricting the possible actions of a program to only those APIs and activities the original code includes, vs C where any machine code can be injected in this same scenario, is a major reduction in attack surface, to me.
One thing to note is that code is data and data is code, in a true, hard-mathematical sense.
The set of
the possible actions of a program to only those APIs and activities the original code includes,
and
C where any machine code can be injected in this same scenario
is exactly the same (unbounded!). Of course it is much easier in practice to effect desired behavior when you can inject shell code into programs, but that’s hardly required. You don’t need to inject code with ROP either (of course ROP itself is not possible in Rust because of other mitigations, this is just an example).
Please note that in no way I am suggesting that Rust is doing anything bad here. Rust is raising the bar, which is great. I want the bar raised even higher, and we know for a fact that this is possible today both in theory and practice. Until we raise the bar, I want people to understand why we need to raise the bar.
At the end of a day you either are type safe or you aren’t. Of course the specifics of what happens when you aren’t type safe depend on the language!
PS: arrays can contain other things than integers, e.g. they can contain function pointers. Of course you can’t confuse an int with a function pointer, but using the wrong function pointer is pretty catastrophic.
In my mind, there are two differences between this code in Rust and similar code in C:
This only allows the user to launch nuclear missiles; it does not allow them to, say, write to the harddrive or make network calls (unless one of the actions contained code that did that ofc); in C, I’d likely be able to make something like this call any system function I wanted to, whether machine code to do that was present in the original binary or not.
In Rust, this doesn’t allow arbitrary control flow, I can’t make this jump to any function in the binary, I can only trick it into running the wrong Action; in C, I can call run on any arbitrary object anywhere in the heap.
ie. in C, this would let me execute anything in the binary, while in Rust it still has to abide by the control flow of the original program, I thought was the case, anyway.
I think you’re saying this is wrong, can you explain how/why and maybe show a code example if you can spare the time?
Good analysis. You didn’t use the words, but this is a great description of the distinction between stocks and flows: https://en.wikipedia.org/wiki/Stock_and_flow. I wish more people talking about software paid attention to it.
I would also argue that crypto should not change often, like bc. You might add ciphers, or deprecate old ones, but once a cipher is written and tested, there should be very little need for it to change. In my opinion.
For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy.
Have they actually rewrote anything? Or have they instead selected a different crypto library they trust better than the previous one? On the one hand, Rust has no advantage over C in this particular context. On the other hand, they may have other reasons to trust the Rust library better than the C one. Maybe it’s better tested, or more widely used, or audited by more reputable companies.
If I take your word for it however, I have to disagree. Rewriting a cryptographic library in Rust is more likely to introduce new bugs, than it is to fix bugs that haven’t already been found and fixed in the C code. I do think however that the risk is slim, if they take care to also port the entire test suite as well.
In the Cryptography case isn’t the Rust addition some ASN.1 parsing code? This is cryptography adjacent but very much not the kind of different that your point about cryptography code makes. Parsing code unless it is very trivial and maybe not even then tends to be some of the more dangerous code you can write. In this particular case Rust is likely a large improvement in both ergonomics for the parsing as well as safety.
You’ve got a point. I can weaken it somewhat, but not entirely eliminate it.
I don’t consider ASN.1 “modern”. It’s over complicated for no good reason. Certificates can be much, much simpler than that: at each level, you have a public key, ID & expiration date, a certificate of the CA, and a signature from the CA. Just put them all in binary blobs, and the only thing left to parse are the ID & expiration date, which can be left to the application. And if the ID is an URL, and the expiration date is an 64-bit int representing seconds from epoch, there won’t be much parsing to do… Simply put, parsing certificate can be “very trivial”.
Another angle is that if you need ASN.1 certificates, then you are almost certainly using TLS, so you’re probably in a context where you can afford the reduced portability of a safer language. Do use the safer language in this case.
Yet another angle is that in practice, we can separate the parsing code from the rest of the cryptographic library. In my opinion, parsing of certificate formats do not belong to a low-level cryptographic library. In general, I believe the whole thing should be organised in tiers:
At the lowest level, you have the implementation of the cryptographic primitives.
Just above that, you have constructions: authenticated encryption, authenticated key exchange, PAKE…
Higher up still, you have file format, network packet formats, and certificates. They can (and should) still be trivial enough that even C can be trusted with them. They can still be implemented with zero dependencies, so C’s portability can still be a win. Though at that level, you probably have an idea of the target platforms, making portability less of a problem.
Higher up still is interfacing with the actual system: getting random numbers, talking to the file system, actually sending & receiving network packets… At that level, you definitely know which set of platforms you are targetting, and memory management & concurrency start becoming real issues. At that point you should seriously consider switching to a non-C, safer language.
At the highest level (the application), you should have switched away from C in almost all cases.
For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.
This is a fair point. When it comes down to it, whether I am right or wrong about it will only be seen in the consequences of the decision that they made.
Here’s the more thorough analysis you’re asking for: this is cryptographic code we’re talking about. Many assumptions that would be reasonable for application code simply does not apply here:
Cryptographic code is pathologically straight-line, with very few branches.
Cryptographic code has pathologically simple allocation patterns. It often avoids heap allocation altogether.
Cryptographic code is pathogenically easy to test, because it is generally constant time: we can test all code paths by covering all possible input & output lengths. If it passes the sanitizers & valgrind under those conditions, it is almost certainly correct (with very few exceptions).
I wrote a crypto library, and the worst bug it ever had wasn’t caused by C, but by a logic error that would have happened even in Haskell. What little undefined behaviour it did have didn’t have any visible effect on the generated code.
Assuming you have a proper test suite (that tests all input & output lengths), and run that test suite with sanitisers & Valgrind, the kind of bug Rust fixes won’t occur in your cryptographic C code to begin with. There is therefore no practical advantage, in this particular case to using Rust over C. Especially when the target language is Python: you have to write bindings anyway, so you can’t really take advantage of Rust’s better APIs.
These bugs still occur in critical software frequently. It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion), and the safer language guarantees more than your suggestions would. And there’s also no risk of someone forgetting to run those things.
These bugs still occur in critical software frequently.
Yes they do. I was specifically talking about one particular kind of critical software: cryptographic code. It’s a very narrow niche.
It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion)
In my 4 years of first hand experience writing cryptographic code, it’s really not. Rust needs the same test suite as C does, and turning on the sanitizers (or Valgrind) on this test suite is a command line away. The real advantage of Rust lies in its safer API (where you can give bounded buffers instead of raw pointers). Also, the rest of the application will almost certainly be much safer if it’s written in Rust instead of C.
And there’s also no risk of someone forgetting to run those things.
Someone who might forget those things has no business writing cryptographic code at all yet, be it in C or in Rust. (Note: when I started out, I had no business writing cryptographic code either. It took over 6 months of people findings bugs and me learning to write a better test suite before I could reasonably say my code was “production worthy”.)
Rusts advantage goes much further than at the api boundary, but again the discussion should be around how to get safer languages more widely used (ergonomics, platform support) and not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”. To me it is like saying “super careful contractors with perfect tools don’t need safety gear”, except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?
Rusts advantage goes much further than at the api boundary
Yes it does. In almost all domains. I’m talking about modern cryptographic code.
again the discussion should be around how to get safer languages more widely used (ergonomics, platform support)
Write a spec. A formal one if possible. Then implement that spec for more platforms. Convincing projects to Rewrite It In Rust may work as a way to coerce people into supporting more platforms, but it also antagonises users who just get non-working software; such a strategy may not be optimal.
not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”.
You’re not hearing me. I’m not talking in general, I’m talking about the specific case of cryptographic code (I know, I’m repeating myself.)
In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.
In this specific case, Rust is not safer.
In this specific case, you need that perfect test suite. In either language.
In this specific case, you can write that perfect test suite. In either language.
except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?
I really don’t. I root out potential mistakes by expanding my test suite as soon as I learn about a new class of bugs. And as it happens, I am painfully aware of the mistakes I made. One of them was even a critical vulnerability. And you know what? Rust wouldn’t have saved me.
Here are the bugs that Rust would have prevented:
An integer overflow that makes elliptic curves unusable on 16-bit platforms. Inconvenient, but (i) it’s not a vulnerability, and (ii) Monocypher’s elliptic curve code is poorly suited to 16-bit platforms (where I recommend C25519 instead).
An instance of undefined behaviour the sanitizers didn’t catch, that generated correct code on the compilers I could test. (Note that TweetNaCl itself also have a couple instances of undefined behaviour, which to my knowledge never caused anyone any problem so far. Undefined behaviour is unclean, but it’s not always a death sentence.)
A failure to compile code that relied on conditional compilation. I expect Rust has better ways than #ifdef, though I don’t actually know.
Here are the bugs that Rust would not have prevented:
Failure to wipe internal buffers (a “best effort” attempt to erase secrets from the computer’s RAM).
A critical vulnerability where fake signatures are accepted as if they were genuine.
Lesson learned: in this specific case, Rust would have prevented the unimportant bugs, and would have let the important ones slip through the cracks.
In this discussion, I think it is important to remind that cryptography developers are explicitly and intentionally not writing modern cryptographic code. One thing they want to use Rust on is ASN.1 parsing. Modern cryptographic practice is that you shouldn’t use ASN.1 and they are right. Implementing ASN.1 in Rust is also right.
In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.
I disagree.
In this specific case, Rust is not safer.
I disagree here too.
In this specific case, you need that perfect test suite. In either language.
I partially agree. There is no such thing as a perfect test suite. A good crypto implementation should have a comprehensive test suite, of course, no matter the language. But that still isn’t as good as preventing these classes of bugs at compile time.
Rust wouldn’t have saved me.
Not really the point. Regardless of how lucky or skilled you are that there are no known critical vulnerabilities in these categories in your code, that disregards both unknown vulnerabilities in your code, and vulnerabilities in other people’s code as well. A safe language catches all three and scales; your method catches only one and doesn’t scale.
Note that I did go the extra mile and went a bit further than Valgrind & the sanitisers. I also happen to run Monocypher’s test suite under the TIS interpreter, and more recently TIS-CI (from TrustInSoft). Those things guarantee that they’ll catch any and all undefined behaviour, and they found a couple bugs the sanitisers didn’t.
that disregards both unknown vulnerabilities in your code
Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project? Because it seemed like a broader statement originally.
I went out of my way not to scale.
I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.
Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project
I was talking about Monocypher specifically. Other projects do have dependencies, and any project that would use Monocypher almost certainly has dependencies, starting with system calls.
I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.
Fair enough. I was thinking from the project’s point of view: a given project only need one crypto library. A greenfield project can ditch backward compatibility and use a modern crypto library, which can be very small (or formally verified).
Yes, other people write cryptographic code. I myself added my own to this ever growing pile because I was unsatisfied with what we had (not even Libsodium was enough for me: too big, not easy to deploy). And the number of bugs in Monocypher + Libsodium is certainly higher than the number of bugs in Libsodium alone. No doubt about that.
Another reason why crypto libraries written in unsafe languages don’t scale, is the reputation game: it doesn’t matter how rigorously tested or verified my library is, if you don’t know it. And know it you cannot, unless you’re more knowledgeable than I am, and bother to audit my work yourself, which is prohibitively expensive. So in practice, you have to fall back to reputation and external signs: what other people say, the state of documentation, the security track record, issues from the bug tracker…
Why make a choice which prevents an entire class of bugs when you could simply put in extra time and effort to make sure you catch and fix them all?
Why lock your doors when you can simply stand guard in front of them all night with a baseball bat?
While personally would back the cryptography devs’ decision here, I think there is a legitimate discussion to be had around whether breaking compatibility for some long-standing users is the right thing to do. This post isn’t contributing well to that discussion.
I can’t help but wondering why obviously bad edge-cases like this come to exist in the first case. The surface level reason is ‘the scoping model is the problem’, but how does one not understand that such a scoping model is an obvious footgun to begin with (even if the specific issue mentioned here is missed)? C’mon folks, ML and many others got these things right in the 70s. All you needed to do was copy ML. Don’t make obviously terrible language design decisions.
Maybe I’m being unnecessarily harsh: the last decade has brought brilliant progress in language design and perhaps there’s a big hindsight bias here that made this issue less clear during Go’s inception. Still: ideas like ‘hey, maybe don’t let locals outlive the scope in which they were defined’ seem obviously sensible and are generally easier to implement when doing compiler development anyway.
Go chose to have
append, modelled after libcrealloc. It has pervasive multi-threading and non-thread-safe built-inmap. So in general, Go is not that focused on preventing footguns.What is the footgun with append/realloc?
You have to use it as:
This boilerplatey reassignment is crucial. If you forget about the reassignment, or assign to a variable that is a temporary/scoped copy of a variable, rather than the place that owns the slice:
then it will sometimes seem to work, until it won’t, depending on capacity.
It’s a very low-level design that lays bare the reallocation happening. It’s fine to have it in the language for advanced uses, but I’d expect language to have an everyday foolproof alternative that ensures the
slice’s data pointer is always updated.And the other shoe is that
appendlooks like it copies the slice so some folks will writes2 = append(s1, item)to create a copy-with-append, this is always an error because if s1 has sufficient capacity they will now be sharing backing buffer, and will overwrite one another’s data.I find your view fairly harsh, but I can see why you and others may not, and the degree to which it’s “unnecessary” is debatable. Perhaps your view is fair and justified.
My opinion on, “why obviously bad edge-cases like this come to exist in the first case” is simple. People make mistakes, and language designers are people. And while this mistake may seem obvious with hindsight, I don’t think it’s necessarily obvious in the absence of it.
I am no language design expert so maybe I am missing something. But if we don’t let locals outlive the scope in which they were defined, how would closures work at all? Closures may be called after the scope has exited and the closures still need the variables in the scope that has exited to work correctly, don’t they? What am I missing?
‘Scope’, when it comes to language semantics, doesn’t mean ‘lifetime’.
It’s fine for values to outlive the lifetime of their scope via closures (provided you have safeguards in place to prevent dangling pointers such as a borrow checker, GC, or whatever)
Scopes are generally considered to be ‘naming contexts’: regions of code in which a particular name corresponds to a specific value (or is accessible at all).
I think it’s expected, by most programmers, that the start of a loop body denotes a new scope on every iteration, and that elements from the last iteration probably shouldn’t carry over unless passed out into an enclosing scope as a side-effect (i.e: mutability, or a similar such mechanism).
That’s not to say that the same memory cannot be reused by each iteration, but that previous state should not be observable.
If the loop variable itself is declared outside the loop body and its value persists between loop iterations (like i++) then it makes sense to me that it is a single variable in a parent scope separate from the loop body.
Not everyone who makes a programming language has experience with every other programming language—not everyone who makes a programming language is even a programming language enthusiast. The Go folks clearly built a language aimed at scratching their own itches and they mostly got it right. Most languages fail to do the most important stuff well (at least as it pertains to software development anyway) because they fixate on minutiae like this. I guess I’m glad Go fixed this, but it’s not like I’d base my choice of language on it.
Lots of unfortunate practices here. I wouldn’t use this post as an example of how to do things properly.
Spent 4 hours last week fixing our CI to be able to install some modules again.
Just one more page in the saga of Python’s strange, convoluted, and always-broken package management ecosystem.
This is literally an attempt to prevent people from breaking their installs, by shutting down one of the most common ways that happens (installing packages into the “system” Python).
I’m honestly not even sure at this point what would make people happy – if something doesn’t get fixed, people complain it’s broken, and if it does get fixed people complain that it’s a change from previous behavior.
The Python packaging ecosystem is so busted, fragmented, and incompatible that people are breaking it all the time by doing innocuous things.
They release a “fix” for this which breaks my own Python packaging environment in the process, taking me a ton of time and effort to set up a virtual environment and port our service to work inside of it with Docker etc.
Quite a sad state of affairs
pytnon3 -m venvand sourcing the resultant activation scripts as part of the CI setup has been the right answer since dubstep was cool…What’s sad about ensuring nobody has to struggle with what we did ever again?
Couldn’t you have told me this when dubstep got cool ? Finding out about this now, after just going full on pip install..
Well, if it’s any consolation, dubstep seeps poised for a comeback, so in a way, I did ;)
The Python packaging ecosystem has standard tools that work quite reliably. The main problem is people who refuse to use them, build their own rickety Jenga tower of stuff as an alternative, and then blame “Python packaging” when it doesn’t work.
For years now,
piphas tried to detect this kind of situation and issue a warning reminding you that installing into the global “system” Python is a bad idea. And every best-practices guide for how to package and containerize Python that I’m aware of has recommended avoiding installing into the “system” Python and instead creating a venv.But finally, distros who are tired of getting bug reports from people who did this and broke their system are just enforcing a blunt “nope, you can’t do that”. As I see it, you’ve had ample warning and ample time to learn about and adopt the clear recommended best practice (which is not some sort of constantly-changing thing – “use a venv” has been the advice for years and years now). I don’t see how it’s Python’s fault that you refused up until now.
There’s literally a website dedicated to listing the dozens of different options for just package management: https://chadsmith.dev/python-packaging/
That dude who founded Sentry just released a new one like 2 weeks ago as well and my Twitter is full of people all hyped about it.
I’m not even trying to say that this fix from Debian totally disabling pip is a bad thing. I’m just complaining about how hilariously messed up the whole Python packaging ecosystem is and what a pain it is to work with it.
Which doesn’t contradict what I said. There are in fact standard reliable tools:
setuptoolsas the package build backend,pipas the package installer/uninstaller, andvenvas the isolation mechanism.And like I said, a huge percentage of perceived issues with “Python packaging” are people deciding to go off and build their own thing instead. At which point the pain they deal with is entirely their fault and nobody else’s, because the standard tools really do work and are not hard at all to use.
Really it’s like if people insisted Rust packaging was some horrid hellscape and the moment you asked them what problems they have with Cargo they’re like “Cargo? I don’t use that…”`
I was just using the standard tools (pip) and they broke unexpectedly and took a good amount of effort to fix.
People have built these myriad alternatives because they have gripes with the default tools, like pip installing stuff with root permissions at the system level, or virtualenv causing 8 different identical 2GB copies of Tensorflow to get installed on my system in various random subdirs, etc.
It’s very strange that you choose to defend Python packaging of all things. It’s notoriously bad; people make memes about it. And in the ~8 years I’ve been programming, it’s gotten worse if anything - not even counting the 2->3 transition that people are still paying down the costs of like 15 years later.
You may have been using standard tools, but you weren’t using them correctly for whatever reason.
Now you are forced to either use them correctly (which is a good thing) or add a single flag to keep using them incorrectly at your own risk.
Not really rant worthy.
I think a part of a problem/perception is that there is quite a lot of blogposts each advocating different approaches. When I was searching some time ago, almost nothing pointed to authoritative sources (at least not the top results where from python.org or pypa.org). When I looked for an inspiration how my colleagues approach building/testing/publishing a python package/app, I was again met with thousands ways of entering the world of hurt — because misinformation spreads easily and is hard to fix.
4 hours? And you don’t have to deal with the consequences of folks doing ‘sudo pip install’ ever again? And you’re … complaining?
It’s a Docker container. It is the whole universe as far as packing is concerned. The fact that I have to create a virtual environment inside my Docker container so I can install a Snowflake Python library is ridiculous to me.
It’s a Docker container. It’s an entire isolated operating system, and it turns out a lot of Linux distros use and rely on tools written in Python, which makes messing with the modules available to the default system Python a very bad idea. So you use a venv, which is the standard recommended thing and has been for years.
Heh, this is ironic, no? Docker containers are there to provide packaging, isolation, and distribution of apps. The fact that many use them as a replacement for a VM just shows people don’t follow the best practices. Just the same as with pip…
If one uses docker containers in the right way, having a venv inside of it really feels wrong, because the container should be an unit of isolation.
Like I said, the issue is that Python is both a language people want to write applications in, and a language distros want to write system tooling in. Nobody would be surprised if you told them that replacing key
.sofiles in a system shared location might break things. Why are people surprised if you tell them that replacing key.pyfiles in a system shared location might break things?So the Docker container is not, in itself, a sufficient unit of isolation for Python code, and never was. Which is why creating a venv is such a strongly recommended practice (and now enforced by some distros).
The fact that distro managed packages for language libraries is a thing at all seems ridiculous to me. It’s a losing game. Some libraries end up specially anointed because they have a maintainer for some distro, some dont, there’s no relation between the annoited ones and the good ones*, and the distro ones are always out of date. It puts languages in a weird place because they want to support their own packaging ecosystem, and also interoperable with a ton of distros, which is a drawn and quartered specification. So like, either distros manage packages and anything else you have to install from source (hope your distro has the right build system and compiler, resolve your deps manually), or languages have environments they can self manage. NPM spearheaded this wacky idea of “why not both” in which if you sudo it, it will futz around in directories that really should be distro managed to fake up a “global install” concept, and python followed. This has since been regarded as a bad idea, and this is the band-aid rip. It never was what it pretended to be, you got lucky if it ever worked for you, I haven’t used snowflake, maybe they figured something out I didn’t see anywhere else.
Edit: I don’t know how to make the asterisk not a bullet :/
I find that the header file problem is one that tup solves incredibly elegantly. It intercepts filesystem calls, and makes any rule depend on all the files that the subprocess accesses. Solves headers in an incredibly generic way, and works without requiring hacks like -MMD.
Not sure if the author is here, but if you are, any plans to support something like that?
So the “proper” way is to intercept the filesystem calls in a non-portable manner and depend on anything the program opens without regard for whether it affects the output or not (like, say, translations of messages for diagnostics). While explicitly asking the preprocessor for an accurate list of headers that it reads is a hack?
The problem with the second option is that it isn’t portable between languages or even compilers. Sure, both GCC and clang implement it, but there isn’t really a standard output format other than a makefile, which isn’t really ideal if you want to use anything that isn’t make.
It’s an unforunate format, but it’s set in stone by now, and won’t break. It has become a de facto narrow waist with at least 2 emitters:
and 2 consumers:
Basically it’s an economic fact that this format will persist, and it certainly works. I never liked doing anything with it in GNU make because it composes poorly with other Make features, but in Ninja it’s just fine. I’m sure there are many other non-Make systems that parse it by now too.
That’s a fair point, also didn’t know Ninja supported it but it makes sense. I wonder if other languages support something similar to allow for this kind of thing, though many modern languages just sidestep the issue all together by making the compiler take care of incremental compilation.
Most tools could probably read the -M output format and understand it quite easily. It doesn’t use most of what could show up in a Makefile - it only uses single-line “target: source1 source2” rules with no commands, no variables, etc. I imagine if someone wanted to come up with a universal format, it wouldn’t be far off from what’s already there.
But.. don’t you want to update your program when diagnostic messages are changed? The FUSE mount doesn’t grab eg. library and system locales from outside the project root, so it only affects the resources of the project being built[1]. Heaven forbid you’re bisecting a branch for a change that is, for reasonable or cursed reasons alike, descended from one of those files..
For those interested, I’ve pitched tup and mused about this in a previous comment here.
[1]: Provided you don’t vendor your all dependencies into the repo, which I guess applies to
node_modules! Idk off the top of my head if there’s a way to exclude a subdirectory for this specific situation, or whether symlinks would work for controlling the mechanism.Edit: Oh, it’s u/borisk again! I really appreciated your response last time this came up and hope you’re doin’ great c:
Edit 2: Oh, and you work on a build system! I’ll check it out sometime ^u^
I originally started Knit with the intention of supporting automatic dependency discovery using ptrace. I experimented with this with a tool called xkvt, which uses ptrace to run a list of commands and can generate a Knitfile that expresses the dependencies. However, I think this method is unfortunately more of a hack compared to -MMD because ptrace is non-portable (not well supported/documented on macOS and non-existent on Windows) and has a lot of complexity for tracing multithreaded processes. A Fuse-based approach like the one used by Tup is similar (maybe more reliable), but requires Fuse (a kernel extension), and also has the negative that automatic dependency discovery can sometimes include dependencies that you don’t really want. When I tried to use Tup for a Chisel project I ran into problems because I was invoking the Scala build tool which generated a bunch of temporary files that Tup required to be explicitly listed as a result.
I think if Knit ever has decent support for an automatic dependency approach, it would be via a separate tool or extension rather than directly baked into Knit by default.
I’ve kinda done something like this over the years, only less purposefully. And I thought I’d settled on Nim, but I rage-quit it a few months ago after a particularly egregious “it’s my language and I can put in any footguns I want*” decree by the Benevolent Dictator-For-Life on the forum.
Anyway for now I’ve ended up at Swift. (And yet I keep coding in C++ because Reasons. Sigh.)
I wonder why the Swift version is so slow. Arithmetic is overflow-checked by default, and arrays bounds-checked, but the OP said they turned that off. Maybe using (ref-counted) classes where structs would have sufficed? Or passing arrays around in a way that defeats the COW optimizations?
* BEGIN RANT There’s a newish Nim feature that lets you declare references as non-null, and uses control-flow analysis to prevent you from assigning/passing a null value, at compile time. Awesome! Only, the analysis was failing in lots of my code and complaining a variable might be null, when I had just checked it right above. After I made some reduced test cases and reported the problem, the BDFL told me (brusquely) that the control flow analysis ignores “return” statements. And that this is not a bug and will not be fixed … because you should be assigning to “result” instead of early-returning (praise Wirth!) This despite “return” being a perfectly cromulent part of the language that’s been around for ages. At this point I decided a single BDFL, esp. a cranky German one [I’m from a German family myself, I know the type] is probably a misfeature and I should look elsewhere. END RANT
Do you have a link to the forum discussion?
This must be what he means – but it comes up all the time. There must be a dozen github issues/RFCs/forum threads where it is referenced. Araq is so strongly opinionated about this one that it is surely a “backing” example to his line in the Zen of Nim: If it’s hard for the compiler to reason about then it’s hard for people to and so you should not do it.
While I love Nim, I disagree on this line item. I think people can be bad at reasoning in situations where computers can be good, and this can matter to program notation - like closing parentheses in Lisp vs. indentation where humans are just bad at “base-1/unary” after about 3..4 things (hence the 5th diagonal slash after ||||, as another example). Even adamant Lispers will say “Let your editor indent/do that for you!” - a paraphrase of saying “a program is better at counting” - they just recommend a different program from the compiler. ISTM, early return is a similar case (but more in the technical weeds).
That is really discouraging for a language I’ve had a lot of faith in. Thanks for sharing.
I’m sorry to hear that you fell out of love with Nim. I always enjoyed hearing your perspective on the language.
If you want to tinker, this looks to be the Swift source. I reproduced the 1.5 hour running time estimate by commenting out line 364’s call to verbose printing function
output_state_choice. Commit history suggests this was left out during the test. Despite some references to the number of cores in code, I found it used just one core, though I don’t know how the other implementations behaved. Memory grows steadily for the at least the first minute or two, so you could be onto something with copy-on-write behavior.For me, the Nim hits ~590% utilization (if 1core=100%). I boosted NUM_THREADS from 6 to 16 on a 16-core and that util didn’t change. So, making the work partitioning more fine-grained could maybe yield a 10x better time on a 64-core AMD 3990X – depending upon contention, of course. { Famous last words, I know! :-) }
The beauty of open source is that Nim can be forked.
Having and maintaining your own private language seems like a bad idea. And unless you have a LOT of free time and some very good ideas, trying to attract supporters away from an already-niche language seems like a bad idea, too.
I disagree. If one or two central people that maintains an open source project are not easy to cooperate with, then it can be very fruitful over time if someone forks it.
Also, forking a project does not necessarily mean that a single maintainer needs to do all the work. Receiving pull requests does not need to be that time consuming.
In addition to this, some forks can be maintenence/“LTS” projects, they don’t have to keep the same pace of development to be useful. Sometimes a few selected patches going in can mean more to users than a continous stream of features.
You’re welcome to D. The language is awesome. However, such “BDFL” have teratons of focused complaints to answer about so it’s not necessarily a good idea to escalate the problem on the internet instead of being patient and help fix it.
You’re dropping the whole language because of one extremely niche feature 99.9% of developers would never stumble on? You know it’s not the language that looks bad in this story right?
I don’t think you understand the feature he’s complaining about correctly, because it seems to me to be very common, as attested by @cblake’s comment that “it comes up all the time”.
There’s cross-talk here. The specific A) strictNotNil feature supporting early return is (likely) a small niche, and B) early return/structure in general is much bigger (& what I meant by occurring regularly, e.g. here). Lest quick/casual readers be confused, early return/break are absolutely accepted {“cromulent” :-) } parts of Nim - not going away. Araq pushed back on hard work for A) he feels other PLs skip while being overwhelmed to get Nim2 in shape (& did not write a misleading doc in dispute according to
git).@snej’s earlier comment (& that Forum thread) indicate he was ok with incompleteness & improving docs. Dropping Nim was more related to a “cranky single German BDFL” - a feature of a community, not a programming language. (I agree “completely and objectively wrong” was needlessly inflammatory rhetoric, but “put in footguns” is also.) Anyway, like @agent281 I am also sorry to see him leave!
These disagreements are often about division of labor & perspective taking, not “objectivity” (esp. given the current state of human psychology as a science). To connect to my prior example, Lispers also complain about offside rules making macro writing “objectively” harder at a cost of “objectively less readable” syntax. Both compiler writers & language users understandably guard their time, driving much emotion.
I honestly never went back to look at that thread after my last reply. I probably won’t.
Maybe I’ll try Nim again sometime. I turned to Swift on the rebound and wrote a few thousand lines over the holidays (not my first waltz with that language) and largely enjoyed it except for the incredibly awkward APIs for working with unsafe constructs. (I wasn’t bridging to C, just marshaling data for a binary network protocol.) Which is to say I’m not totally happy with Swift. I’m still waiting for the perfect language.
It was the straw that broke the camel’s back. And where did you get the idea that null checking or the “return” statement are extremely niche features? Ok, null checking isn’t on by default in Nim, but it’s basically table stakes in most new languages ever since Hoare’s “billion dollar mistake” went viral.
If a complex condition is always unreachable, I’d prefer to see my compiler or static analysis tool complain about that instead - since, in many cases, having a complex condition that is always unreachable might itself be a bug - rather than treating the condition as correct and the unreachable statement as dead code.
Of course in certain cases - like ‘0 && expr’ or ‘1 || expr’ (including with macros), those kinds of warnings should be suppressed - but in cases like the example in the article, I personally would rather see a warning about the condition itself being suspiciously constantly false.
Ruby does this right “open do..”. Python does it right, “with … do”. Any reasonably wrapped smart resource will catch this at compile time; we used to do this for memory and files in the ’80s. ‘defer’ is just a way to make mistakes.
Don’t these still require you to opt-in to the correct behavior? You can just forget to use those constructs too.
openis the classic example where not usingwithwill still allow you to do your work. But most libs will dowith thing() as actual_object: .... If you doactual_object = thing()then you’re not getting the object you want but just a manager object.The way to get around using
withis doingactual_object = thing().__enter__(), which is a bit unwieldy.Ah ok I guess the title says what it literally is but I was hoping it compiled to Go or something. It’s just written in Go and generates code with LLVM. No Go interop at all. Slightly disappointing (only because I’m predicting the next hit language to be an ML on the Go ecosystem a la F# and Scala) but an awesome compiler project nonetheless.
I would also love to see an ML derivative that compiles to and interoperates with go. I feel like that would be the perfect language for me.
I agree with this, I think Go is a useful target; I may be influenced by the fact that I’m not much of a fan of writing Go as well, but I do think it’ll be a useful target. I spent a fair amount of time getting carML to output Go nicely, and it’ll be something I target in coastML for that reason as well.
I’ve had this idea for quite some time, but not a ton of motivation. I feel like this is so close that I should be jumping up and down to do it… but, time. Probably won’t be me.
Couldn’t the last example be reduced to:
For a similar expected behavior?
The
one == endis kind of redundant with*end != '\0'.Might not be perfect, but API-wise this is not as smelly as it is presented, in my opinion. Actually with C limitations it kind of makes sense. Any parsing error due to numbers that cannot get represented in a long are passed into
errno, which is a common way to represent errors in stdlib. Using theendpointer seems like a reasonable and very flexible way to let the caller deal with what should be considered an invalid string. In some case, someone might not care about trailing character (so could useend == bufinstead of*end).Not necessarily. If the string is
"", then we would haveone == end, but not*end != '\0'.Good point didn’t thought about this case!
Not quite; you have to check the return value before you can rely on errno.
SLIME, as the expression of roughly 50 years of parallel evolution and refinement of a programming language, developer toolchain, and ecosystem, is absolutely the gold standard. It’s also had 50 years to get there.
Let’s furthermore not forget that homoiconicity and image-based development has meant that your source code, object code, parser, compiler, debugger, and REPL all ship as part of a single mutually-referential whole. That in turn lets you do exactly the kind of whole-program analysis that the RA article bemoans as largely intractable in Rust-land.
C as a language lags by at least 15 years, and Rust is only a smidge over 10 years old. If you want to find something closer to the Lisp/SLIME universe it’s probably in the JS or .NET ecosystems…neither of which is actually fully self-hosting AFAIK, and so you still hit a hard wall of abstraction when you drop below the standard syntax and libraries.
If we’re very lucky, perhaps Rust + $EDITOR[1] circa the ~2040 edition will have achieved something like the smooth integration and workflow that SLIME devotees have been (rightfully) crowing about since at least the late 90s. :)
[1]: I still don’t see VS Code being this One True Editor, but neither Emacs (given the Lisp-ness discussed above) or (N)Vim seem likely to naturally fill the gap either, since they aren’t born of the same zeitgeist and aesthetic as Rust. Perhaps it’s no coincidence that so many folks in the Rust community are focused on text processing and GUI frameworks, and the ideal “Rust environment” is both imminent and inevitable.
Forgive me, I don’t know slime well, but wouldn’t it have the same problem? Assume some macro expands its arguments by reversing the names of symbols. How would slime know to autocomplete the reverse of what is intended?
This is why Rust’s ecosystem is fundamentally and unrecoverabky broken and which is why I’m glad to be able to rely on my package manager and large standard library in C.
Probably the only reason why Rust crates aren’t pwned to oblivion is the fact nobody cares about Rust other than evangelists and those who still have to find out how messy it is.
To provide an alternative, look at Ada, an absolutely beautiful and secure language with many great ideas and concepts.
Okay, I don’t really care about Rust, but I doubt this. If I were to hazard a guess, I’d say that the real reason is more likely that executing effective attacks this way is complicated, it’s still a fairly new area, and the Rust community is waking up and mitigating some of the flaws in what is undoubtedly a flawed system.
If nothing else, there is a lot of money to be made successfully exploiting Rust, given the number of high-profile projects now adopting it. Like it or not, Rust got past the ‘only evangelists and idiots’ phase a long time ago, and the teams using it are, with the best will in the world, not going to rewrite their code in Ada. I don’t see the point in writing it off like this.
I care about Rust and I’m neither.
It’s a little bit easy to just point at the shortcomings of one side but ignore the other. You’re trading a mature ecosystem that allows git commit tagging to something that doesn’t even have a definition of external library except for linking with what ever is currently in your path variables. Which is neither secure nor reproducible.
What you’re suggesting is that we should all go and rely on your package manager not to break or get pwned (which did happened multiple times in debian). This kind of build system has no idea of versions, security updates, revocation and such, it just ships what happens to be the thing that got into your package management, together with all the changes they thought are useful. (And sometimes, they actually make it worse.)
There is much to be said about securing build scripts and macros, but most of the things shown in this article also work with c (make scripts, macros, init functions..). And due to all the shortcomings of C it is even easier to hide something in there. When you use what debian ships for TLS, you hopefully trust the debian people (and transitively the authors) and you trust people building your software to have an up to date version of it. When you pin stuff in rust, you trust the people running crates.io (see debian) and the authors that put stuff up there. But your users (or upstream distro) will have to actively select a bad version when building it, to get the same effect. Pick your poison.
In a perfect world we sandbox any build scripts and macros and use the rust-sec extensions for crates.io to verify the trustworthiness of our dependencies. At that point the only thing that can go wrong is a malicious build, built inside your sandbox, delivered to everyone.
To catch issues like this sooner, I have automated your manual step. Instead of running a manual dry-run, looking through it for unexpected changes, and then running a manual real-run, I have my backup script email me the list of files that changed since the last backup as it does the backups, no manual step required. This lets me peruse the email when I have a free minute or two, and it doesn’t take long since this happens daily and the files that SHOULD have changed are fresh in my mind. And if I notice anything suspicious I can go check things out and restore from a previous version if anything is amiss. Seems like the right balance to me.
That’s a nice idea, thank you. Various cron jobs already email me from my servers, but it didn’t occur to me to try to email myself from the iMac. That would be easy enough and, as you say, a better balance between automated and manual.
Java is much older than Rust and carries backward compatibility weights. I am surprised it is not much slower.
C++ is much older than java and I’d be surprised if it wasn’t in the same performance ballpark than Rust for optional. Old age is not an excuse for bad design.
Are you sure it is bad design? Perhaps Java’s design just makes different trade-offs for performance (like consistency or lower mental model overhead or… I’m sure there could be many).
I can confidently say that it isn’t for consistency or mental overhead when compared to c++‘s generic implementation. Else this mess wouldn’t exist: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/function/package-summary.html and requiring fall back on external libs if you want more arguments, like https://projectreactor.io/docs/extra/3.2.0.M3/api/reactor/function/Consumer8.html
I can confidently disagree, having seen the C++ standard library implementations of gcc, llvm, msvc, and a few others.
why would the stdlib implementation matters ? what matters is that as a C++ dev I can just do optional<int>, optional<float>, optional<std::string>, optional<whatever_type> while in Java I have to remember that I must use OptionalInt, OptionalLong, OptionalDouble, and Optional<T> for everything else.
Anecdotally, I find libc++‘s implementation (https://github.com/llvm/llvm-project/blob/main/libcxx/include/optional) fairly readable, outside of the necessary __uglification of headers (and when one takes into account all the various optimizations that it does that Java’s does not seem to do - if I read https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/Optional.java correctly, it means that Java’s Optional does not work with value types since it relies on value being nullable ? how is that remotely serious)
What matters is that in rust I can do
Option<NonZeroU32>and have a type with sizeof 4. C++ can’t do that - how is that remotely serious.I hope this will help you see your replies in this thread in a different way :)
? Of course it can, boost::optional did this for e.g. storing optional<T&> in a pointer for years. Sure, it’s a bit of work to specialize std::optional for your NonZero type but there’s nothing technically impossible
Not really. Sure, you could maybe handcode specializations for a few selected types? Maybe? It’s not a trivial template. But even then this only handles some explicit/short list of types. In rust this is automatic for any type with ‘spare’ bits and works recursively (https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=004c4172a53ba7ab00fc7afb5e3adceb).
Also, since two posts before you complained about java needing external libs then don’t use boost as an excuse for c++ standard libs ;)
If you were implementing your own generic libraries instead of just consuming them, then your view might be different.
For the language designers, both viewpoints are important.
Having had to do this in Java, C# and admittedly mostly) C++, I really prefer the C++ way, especially nowadays with concepts.
Concepts help, but they are quite recent. Also, C++ has objectively quite a few more footguns than the other languages, despite individual preferences.
Java has a huge boat-anchor, the JVM. It’s stuck with a bytecode instruction set, binary format, and runtime model designed in the mid-1990s before the language had even seen serious use. There have been minor changes, like clarifying the memory concurrency model, but AFAIK no serious breaking changes.
This makes some improvements, like stack-based objects, either impossible, or achievable only with heroic effort by the JIT and only in limited circumstances. (See also: the performance limitations of Java generics due to the requirement of type-erasure.)
User beware. The hand-rolled recopies showcased here are better than your average hand-rolled recopies I see on many projects, but they still leave a lot to be desired. For example
mkdir dir; cp foo diris not nearly as robust asinstall -D -t dir foo. But that’s trivial compared to the bash specific syntax in anshfile that should be plain POSIX, or the fact that the simulated argument handling doesn’t work quite the way as the real./configurescript generated by autotools does.As ugly as it can be sometimes, using
autotools(autoconf,automake) is far more robust than hand-rolled scripts will ever be precisely because it has such a long history and wide usage all the edge cases not considered by any one person are already taken care of. And in the end the amount of code you actually have to write and maintain would be less for the examples given.IME many autotools-using projects have failed to build for me, and writing a 12 line makefile by hand could build the same project fine.
2 points.
That is almost certainly not autotools fault, but its so confusing to get started using that it doesn’t always get setup right, and that can certainly make it break for users.
Your 12 line makefile may work for you, but it likely doesn’t work for systems that are not like yours. It probably leaves distro packages writing their own build routines and different OS layouts hacking around the over-simplified model shown in 12 lines. And I say this as someone who wrote THOUSANDS of lines of hand-brew makefiles before finally turning to autotools and finding it solved so many of the edge cases I was working around it really was useful.
I’d argue that if it is confusing/hard to use, it is the fault of the tool.
How important is it that a given piece of software be buildable on the maximum number of architectures?
Where is the line beyond which complexity to achieve portability is more cost than benefit? Is there such a line?
Just as important as it is to have access to more systems that x86_64, and run more operating systems than Windows (or, whatever, Aarch64 and macOS). Portability isn’t just about being able to build libpng for OpenVMS in 2021, it’s also about being able to build whatever is going to be important, under whatever operating system is going to be important, in 2025.
You could presumably drop support for the former without causing much of a fuss, but
autotools& friends allow us to support any number of architectures, whether maximum or not. If we start to package everything with 12-line makefiles we might as well all buy x86_64 laptops running whatever Linux distro op is running and give it up. Then in 2031 we can all gather here and whine how the next best operating system supports a direct-to-brain interface but is held back by the fact that most software only works with Ubuntu and Docker (or, realistically, Windows and macOS…)I don’t think I understand your point. I agree that it’s important to support more than a single platform. But my claim is that there’s diminishing returns beyond a certain point. Do you dispute that? If not, where is that point? I guess I’m claiming it’s somewhere around the top 5-10 in terms of userbase. Do you think it’s more than that?
My point is two-fold (sorry, it may have been useful to be a little more clear).
First, “the maximum number of architectures” is just another way of saying “any architectures one needs”. Any system, no matter how useful, including Linux, started by not being even close to the top 5-10 in terms of userbase. If you restrict your build system’s capability to supporting only the leading two or three platforms of the day (realistically, a short Makefile will at most get you Linux, BSDs, and maybe Cygwin, mingw & friends need quite a few hacks for non-trivial programs – 5-10 is already way more than you can realistically manage for non-trivial software), you’re dooming it to obsolescence as soon as the next different enough thing hits the market.
I bet that most of the programs that use
autotoolsdon’t actually run (at all, let alone well) on all platforms that autotools can build them on. That’s only as big a problem as the users of said software make it to be though, presumably nobody is losing too much sleep over not being able to run qemu on 4.4 BSD. However, if and when you need, you’re at least gonna be able to build it and you can go right ahead and do the substantial porting work you need. Clunky though they may be, programs built usingautotoolsare gonna compile on Fuchsia 3.1 for AR Workgroups or whatever pretty much out of the box (barring any platform-specific things that need code, not build system incantations, of course). Beautiful, minimalistic, hand-rolled, organic Makefiles are gonna be a curse as soon as Ubuntu 20.04 LTS will be a distant memory instead of something that was pretty cool five years ago,It’s not like we haven’t seen this already,
autotoolsbecame a thing because, terrible though it may have been even then (like, m4, seriously?) it was still better than managing Makefiles for even two or three commercial Unices (which meant way more than just the OS back then – also different libcs, different C compilers and so on). It didn’t come up in a vacuum, either, it was based on (or rather tried to improve) a bunch of prior art (Metaconfig, imake).Second, and strictly regarding the number of platforms: in my experience, while the tooling required to build on 1-2 platforms vs. 5-10 platforms is massively different, past that point, the complexity of the tooling itself is pretty much the same, whether you support 5-10 platforms or 50-100. The number of platform-specific quirks obviously increases with the number of platforms, but extending a tool that already supports 10 platforms so as to allow it to support 100 is largely an exercise in implementing platform-specific hacks, whereas extending the tooling (“12-line makefile”) to support 10 platforms instead of 2 basically amounts to writing a new build system.
So while it takes a lot of effort to go from supporting 1 platforms to 5-10, there’s not that much effort in ensuring your software can be built on more than that. If you’ve done that, you might as well go all the way and use a system that has the traction and popularity required to work everywhere for the foreseeable future.
autotoolsisn’t the only one. CMake is another, for example. It’s terrible but lots of popular software is built with it. Anyone who wants their platform to be relevant will have to ensure CMake-built software runs on it, otherwise they’re gonna be stuck without some leading DBMSs, web browsers and so on.I think I understand your point here. But I think the difference between our positions is that you’re judging autotools by its asserted capabilities, and I’m judging it by it’s practical drawbacks. In my experience autotools fails more often than it succeeds, and it delivers net-negative value to projects that use it. I’m sure your experience is different!
I don’t think I buy this. Software is a living thing, and if my project is important enough to stand the test of time, it will grow build capabilities for new architectures as they become en vogue.
I think so. While it’s definitely far from my favourite build system, I can’t remember the last time I’ve seen it fail. However, it’s so widespread that it’s very easy to be lucky and only encounter it in scenarios where it works fine.
When someone else’s 12 line makefile doesn’t work, it’s clear how to fix it. The same isn’t true for autotools, cmake, or many of the other more complicated alternatives.
tbh I don’t actually agree this.
makeunfortunately is quite difficult to debug without turning to external tools. Honestly, I find both cmake and autotools a lot easier to debug than just a raw Makefile, since they have debugging facilities.That said, I don’t turn to
cmakeorautotoolsuntil I hit the level of complexity or reach where that matters, and most of the time just use simple Makefiles.If it is really 12 lines, it is probably less annoying for distros to maintain their own version than to maintain a command line of
./configure .... If something breaks, it’s much easier to fix a Makefile with 12 lines than the whole auto* setup.Not to mention default rules which are good and can change to fit your platform without you trying or caring.
Absolutely, I always assume pkg-config and other of these sorts of things of course that autotools will try to paper over for you.
Can you build a proper shared library on AIX without using
libtool? I’d like to see your makefile try.No idea, and also no interest from me :) If someone wants to build something on AIX that’s cool I guess, and they probably should be using autotools.
You seem to be implying that using -G and ar aren’t enough. Care to elaborate on why not?
This is worthy of a blog post, but while technically you CAN just go
gcc -shared libhober.so libhober.cand call it a day, there’s a lot of subtleties. Specifically:Your binary will export everything by default because symbol visibility sucks.
Your binary is one architecture only - AIX actually has fat libraries, but not binaries. It works by making libraries
.aarchives, and you can import based on member.To circle back a point, you can also put a file in the archive (
shr.imp/shr_64.imp- it is very tempting to call them shrimp files) indicating what should be imported, and its name to link against (we’ll come back to that later.)Want versioning? GNU fucked it up by coming up with two schemas:
libhober.a(libhober.so.69)andlibhober.so.69(with an implicitshr.o/shr_64.odepending on arch). The former scheme is insane to manage via a package manager, so everyone sane went for the latter.Problem: if you do
cc -lhober, it’ll link againstlibhober.so, notlibhober.so.69. This works, but is subtly wrong (don’t have the devel package installed, what if soname bumps for a reason?), but it’s easy not to catch because it works in development.To work around that, your shrimp file can declare the real name of the library to link against.
This kinda makes sense but is unnecessarily brittle. It does make for a great smoke test of how committed to portability you are though. libtool is like violence, it’s the cause and solution to all of life’s problems.
I mentioned using “-G” and “ar” to specifically create AIX-like shared libraries, which doesn’t suffer from this issue.
Some of the others may be issues if you need some of those features, but I don’t see how that really precludes you from building a “proper shared library” that doesn’t use those features with just the compiler and archive tools and a really simple makefile.
I think most people using the really simple makefiles probably aren’t using
-Gandarto make a shared library (do you think Makefile curators give a shit about AIX?), and now you have to curate your shrimp file for exports (careful, it’s REALLY easy to accidentally end up exporting strcpy if you do all, because strcpy is only a static object with AIX libc :) ). Hell, stuff like CMake barely does it.The original position was that shared libraries on AIX are hard or impossible without libtool - but my position is, a simple Makefile can handle that just fine (if the author wants to support AIX explicitly, of course). Now, using libtool may give you different platforms like AIX automatically, but that doesn’t support the claim that a simple Makefile couldn’t build “proper shared libraries” on AIX, if desired.
Are you sure?
Perhaps it’s changed in 7.2; I remember having plenty of versions like strcpy end up exported in shared libraries naively exporting all, instead of a curated export list or using
nmon objects to generate one.Hi, I’ve just published the first release of this program. It is open source released under the Apache 2.0 license.
From the linked document:
The source code archive and GitHub link can be found here: https://sentido-labs.com/en/library/
Edit: in some machines, the miniz library does not compile with
-Wsign-conversion: you can get it to work by removing that option fromCFLAGSinMakefile. This affects onlycedro-new: bothcedroandcedrocccompile before that error.This looks neat, I have a few comments:
Why does the deferred resource release not use
__attribute__((cleanup))? It can generate code that is correct in the presence of exceptions, whereas the output code here still leaks resources in the presence of forced stack unwinding. Is it just that you’re doing token substitution (__attribute__((cleanup))takes a pointer to the cleanup object, so you can have a singlecedro_cleanupfunction that takes a pointer to a structure that contains explicit captures and a function to invoke). The choice ofautofor this also means that it cannot be used in headers that need to interoperate with C++ - was that an explicit design choice?Similarly on the lowering, binary includes are a much-missed feature in C/C++, but the way that you’ve lowered them is absolutely the worst case for compiler performance. The compiler needs to create a literal constant object for every byte and an array object wrapping them. If you lower them as C string literals with escapes, you will generate code that compiles much faster. For example, the cedro-32x32.png example lowered as
"\x89\x50\x4E\x47\0D\x0A\x1A..."will be faster and use less memory in the C compiler. I’m not sure I understand this comment though:What is ‘the current directory’? The directory from which the tool is invoked? If so, that makes using your tool from any build-system generator annoying because they tend to prefer to expand paths provided to tool invocations to absolute paths to avoid any reliance on the current working directory. I don’t actually agree with the assertion here. On every codebase I’ve worked where I’ve wanted to embed binaries into the final build product, those binaries have been part of the source tree so that they can be versioned along with the source.
It’s a C preprocessor and C does not have exceptions
C has
setjmp(3)and friends as a very raw, low-level exception mechanism. It’s basically the same underpinnings, with a much less developer-friendly interface. Still, real C code does make use of it!And
__attribute__((cleanup(..)))doesn’t work with longjmp. Not even C++ destructors run if you longjmp out of a scope. (Both destructors and__attribute__((cleanup(..)))run when unrolling due to an exception though.)C’s longjmp doesn’t do any stack unrolling, it essentially just sets the instruction pointer and stack pointer in a way which breaks lots of stuff. That’s not to say it’s not used though; I’ve encountered it myself with the likes of libjpeg.
C does not have exceptions, but C often lives in a world where exceptions can be thrown. If your C code invokes callbacks from non-C code, it often has to handle exceptions being thrown through C stack frames, even if the C code itself doesn’t handle them. Writing exception-safe code is one of the main motivations for
__attribute__((cleanup))existing.Is it well defined behavior to throw C++ exceptions across C stack frames? For some reason I thought that was UB.
It’s certainly not defined behaviour, since the C++ standard doesn’t (naturally enough) concern itself with specifying what happens when an exception passes into code written in another language.
In practice, functions in C may not have unwind information and an exception that propagates into them will be treated the same as an exception that (would) propagate out of a
noexcept-attributed C++ function. (Off the top of my head, I think the result is thatstd::terminate()is called).However, it is possible to compile C with unwind info (eg gcc has
-fexceptionsto enable this), and in that case implementations will allow propagating an exception through the C functions. At some point the exception still needs to be caught, which you can only do in C++ (or at least non-C) code, so this is only something that’s needed when you have a call pattern like:Well-defined in what context? In the context of the C specifications, any interaction with code not written in C is not well defined. In C++, functions exposed as
extern "C"impose some constraints on the ABI, but no strong definitions. All (well, almost all) functions that are part of the C standard are also part of the C++ standard. This includes things likeqsort, andqsortin C++ explicitly permits the callback function to throw exceptions.More relevant, perhaps, are platform ABI standards. On Windows, throwing SEH and C++ exceptions through any language’s stack frames is well-defined behaviour and the C compiler provides explicit support for them via non-standard extensions.
Most *NIX platforms use the unwind standard from the Itanium C++, which defines a Base ABI that is language agnostic and layers other things on top. This defines DWARF metadata for each frame that tells the compiler how to unwind. You can write these by hand in assembly with the
.cfi_family of directives. If a frame doesn’t contain any code that needs to run during stack unwind, this information just describes how to restore the stack pointer, what the return address is, and where any callee-save registers that the function used were stashed. The unwinder can then walk the stack and restore the previous frame’s expected register set and recurse. With GCC and Clang,__attribute__((cleanup))emits the same metadata that a C++ destructor would in these tables, so the code in the cleanup function is run during stack unwinding. You can use this to do things like release locks and deallocate memory in C if a callback function that you invoke from any language that has exceptions throws an exception through your stack frame. Note that this doesn’t let you catch the exception. There’s no major reason why you shouldn’t be allowed (from C) to block exception propagation, though if you need this then invoking a callback via a tiny C++ shim that doestry { callback() } catch (...) {}will prevent any exceptions (even non-C++ ones that use the same base ABI) from propagating into the C code.Because that is compiler-dependant, at least for now. I want something that would work where a newer compiler is not available. Also, the current mechanism in
cedroallows code blocks with or without conditionals, which are more flexible than the single-argument function required by__attribute__((cleanup)), unless I misunderstand that feature. I’ve actually never used variable attributes in my own programs, only read about it, so I might be missing something.It wasn’t, the reason was to avoid adding more keywords that would be either prone to collisions, or cumbersome to type. For use with C++, you could write the output of
cedroto an intermediate file which would be standard C. I’ll have to think about it in more detail to see how much of a problem it is in practice.I did not realize that, you are right of course! I know there are limits to the size of string literals, but maybe that does not apply if you split them. I’ll have to check that out.
EDIT: I’ve just found out that
bin2c(which I knew existed but haven’t used) does work in the way you describe, with strings instead of byte arrays: https://github.com/adobe/bin2c#comparison-to-other-tools It does mention the string literal size limit. I suspect you know, but for others reading this: the C standard defines some sizes that all compilers must support as a minimum, and one of them is the string literal maximum size. Compilers are free to allow bigger tokens when parsing.I’m concerned that it would be a problem, because as I hinted above my use case includes compiling on old platforms with outdated C compilers (sometimes for fun, others because my customers need that) so it is important that
cedrodoes not fail any more than strictly necessary when running on unusual machines.Thinking about it, I could use strings when under the length limit, but those would be the cases where the performance difference would be small. I’ll keep things like this for now, but thanks to you I’ll take these aspects into account. EDIT END.
I see, that’s again something I’ll have to consider more carefully. I keep the binaries separated from the source code, but it would make sense to put things like vertex/fragment shaders next to the C source.
Thank you very much for your detailed review.
The attribute takes a function, the function takes a pointer. That’s sufficient to implement a closure. For example, you could transform:
Into something like this:
Now you’ve got your arbitrary blocks in the cleanups. If your compiler supports the Apple blocks extension then this can be much simpler because the compiler can do this transform already.
The best way of doing this is to follow the example of Objective-C and use a character that isn’t allowed in the source language to guard your new keywords. A future version of C may reclaim
autoin the same way that C++11 did, and some existing C code uses it already, so there’s a compatibility issue here. If you used $auto then it would not conflict with any future keyword or identifier.After thinking about it, my conclusion is that you are right, so I have changed the program: now the binary file is loaded relative to the including C source file.
Thanks! What are you currently using this for? The place I would imagine it being most useful is for tiny embedded systems that have a C compiler but no C++ compiler. Firmware blobs that want embedding in the final binary are pretty common there.
Well, today I’m continuing work on source code manipulation tools using tree-sitter, which is a parser generator that outputs C parsers. I started with the Rust wrapper but some of the machines where I would like to run it do not have a Rust compiler, some because of the OS, others because of the CPU ISA.
What I’m doing is exploring how much I can cut out dependencies and remain productive. Dependency hell is manageable for a full-time job, but for anything else which I revisit only occasionally it is not acceptable to get derailed by something that used to work, but they changed it and now it does not anymore, and you can not get back to a previous version because of a tangle of up-/downstream dependencies.
The use case of resource-limited machines like microcontrollers and retrocomputers is also a goal: I hope to resume work in that respect soon; like many people, I have a bunch of such machines lying around waiting for me to find some time for them. The intention is that a simpler build chain should make that easier to do as a spare-time job.
And then, binary includes are very useful to cut down on dependencies even on modern machines: one example is for simple GUI applications, where using nanovg and embedding the fonts and images I can get an executable that does not require installation, and depends only on glibc and the various libGL*, libglfw libraries which works well in practice for me. I find this much easier to keep portable than using big complex GUI frameworks, which I admit provide lots of difficult-to-implement features: for some programs though, I find that my choice is not either minimal dependencies with spartan features or more dependencies with complete features, but either minimal dependencies or a non-compiling/non-running program.
For pretty much anything I was using C for 10 years ago (including a Cortex-M0 with 8 KiB of RAM) I’m using C++. C++17 is available on any system with GCC or Clang support. The C++ standard library is sufficiently ubiquitous that it counts as a dependency in the same way that the C standard library does: it’s there on anything except tiny embedded things. It can easily consume C APIs and with modern C++ there are a lot of useful things for memory management and so on.
I do see your point, and I’ve used C++ for decades and expect to keep using it in the future: the improvements in the last years after the stagnation period have made it much more comfortable to use.
This is really great! I love it!
Thanks, I would like to hear about your experience, positive or negative, once you get to try it out.
Switching to using Lua for Neovim seems like it’ll be really powerful. I’m just a little scared to make that final jump, because there’s way to really go back to Vim after doing this. I’ve already leveraged a bunch of Neovim-only stuff, but this is a lot more extreme.
The Lua support in Neovim is one of the things making me eager to look at it. Lua is a fantastic little language and there’s no reason to be stuck with something like vimscript in 2021.
I looked a bit at lua, and a lot of stuff seemed much more verbose/awkward to me.
I kinda of like VimScript 😅 There are a few annoying parts (line continuations, ugh, and I still forget that stupid
a:sometimes), but overall I find it works fairly well, especially since lambdas (already some time ago) and the method syntax (also some time ago, but never ported to Neovim) were added. It’s a fairly small language that fits its purpose reasonably well.It’s perhaps a bit of an acquired taste though.
Neovim is claiming that lua is much faster than VimScript. I’ve seen VimScript chug on files with just a few hundred lines, so I’m hoping I can convert my heavier functions to lua and get a speed boost.
I can’t say I ever ran in to issues with this myself; most of the times I ran in to performance issues it’s because you do something like
:%s/<something rather complex on a large file>/, but I’m not sure if Lua will really help with that? From what I can tell, most of the performance issues is when people implement stuff like fuzzy matching in VimScript and similar computation-heavy stuff.The new VimScript9 is quite a bit faster (but still slower than LuaJIT), but I haven’t really looked at it.
Wow, this is the first time in twenty years I’ve heard anyone express an emotion about vimscript other than disgust.
if you keep your dotfiles in source control (which I recommend if you aren’t already), you have nothing to lose. If you make the switch and decide later that it’s not for you, just revert back to the previous version of your config that uses vimscript.
That only works until you customize further in lua.
I’m not sure what you mean. Sure, you lose some further custom lua config (until you port it to vimscript), but you won’t end up worse off than you were before you switched if you have your config in source control.
I have recently switched not only to Lua, but to Fennel as well and I am pretty happy with result. Now I need to port rest of my config from VimL to Fennel.
Wow, that’s amazing! Did not know Fennel could be used like this.
do you notice any delay from having to compile the fennel files to lua on every startup?
Slight, but nothing annoying me. In the past I was trying to cut load time as much as possible, but now I stopped giving a damn. I haven’t tried precompilation though.
If you’re using a lisp for your editor scripting.. At what point does evil/emacs becometthe saner choice, though?
As soon as Evil will learn into all my favourite quirks of Vim. I have tested Emacs in the past and there was always some small nitpick that was missing and was irritating me. Now I get the best of two worlds.
Is there a reason that’s a problem? Neovim is a fork of the vim codebase and migrating to it doesn’t seem all that different from upgrading vim and then using plugins that require the new version as their minimum version. Neovim is basically just vim that is making faster progress on cool new features than mainline vim is.
Neovim 0.5 lets you write your vimrc as a lua script, which Vim8 can’t parse.
The big gotcha I hit with Neovim was that it uses a different format for undo files. I have this in my vimrc:
This avoids cluttering my working directories with undo / backup / swap files and gives me effectively unlimited persistent undo, my favourite feature of vim. If I run neovim on a file that I’ve edited with vim, it complains. Similarly, if I edit with nvim and then vim, vim can’t read the undo file. I wish they’d change the name of files if they were going to have things for the same purpose as vim but with incompatible formats.
This can be achieved with something like
I believe the undofile format became incompatible only very recently, when more information was added to undofiles.
That’s what I did, but it’s annoying that I have to.
I was also quite excited by the built-in LSP support, but all of the tutorials I found required incredibly complex configs and didn’t explain how to configure the LSP integration for something like clangd for C/C++.
There are heaps of incompatibilities between Vim and Neovim, and have been for years.
https://news.ycombinator.com/item?id=27719968
What are some neovim features that you’ve really liked? I generally see the same things trotted out by fans, like treesitter and in-build LSP, but these aren’t things I’ve felt like I was missing in Vim (I do use LSP, plugins work well for this). I’d love to hear what benefits you have found, as an experienced Vim user?
It’s a bit of a moving target, because a lot of neovim innovations are eventually implemented in vim. That’s what happened with floating windows, async jobs, and terminal emulation. So I guess the feature I really like is being a couple years ahead of everyone else :)
There’s also a few smaller things: I really like
inccommand, which I don’t think has been ported over yet. And I just found out I can use the lua bindings to do this:I expect that as I get more comfortable with lua and the neovim API I’ll have more say. I write a lot of custom functions and vimL is one of the worst languages I’ve ever used.
EDIT ooh something I just realized I can do:
That’ll make it easy to create setting groups.
No, no
inccommandin Vim still. I was quite keen on this for a while until I discovered traces.vim which is a lot more powerful.Yeah vimL has its quirks. The actual examples you’ve shown would look pretty similar in vimscript though … although they would use
:execute, e.g.exe 'nnoremap' c_key '<c-w>..key. I’ve come to quite like vimL but I realise that puts me pretty firmly in the minority. I hate the line continuations but those are going away in vim9script.It does look like configs will diverge now, to the lua neovim scripts and the Vim vim9script scripts. Which I suppose is good, trying to support 2 diverging editors with a single code-base is annoying, as each add different features.
You can map stuff in loops in regular VimScript:
For a somewhat creative use of this you can see this ;-)
I guess the issue is that a lot of people just can’t be bothered to invest in learning VimScript, which is fair enough. But pretty much anything you can do in Lua you can do in VimScript, and vice versa.
Also, there’s more third party Lua libraries. A while back I needed to fill an autocomplete based on information in a YAML file, but couldn’t figure out how to parse YAML in VimScript. I ended up writing a python script to first convert the file to JSON, then use
json_decode. It worked in that case but would have been nicer to just import a library and read the YAML directly.I think this - the availability of the Lua library ecosystem - is probably a “bigger deal”, looking forward, than the performance/tooling/ergonomics benefits Lua brings over VimScript. I haven’t seen too much adoption of Luarocks packages in plugins yet (though there are some examples, e.g. https://github.com/camspiers/snap), but I expect this to change now that 0.5 is stable.
I will also say that, while anything* you can do in Lua you can do in VimScript, I personally find it much easier to do most things in Lua - partially due to familiarity, but also due to the simplicity of the language and the relative lack of footguns, compared to VimScript.
*In Neovim specifically, Lua provides access to luv, which I’m not sure you can access from VimScript. In my experience, having the
libuvevent loop and utility functions readily at hand makes it possible to write functionality which would be at least substantially more painful without these tools.Why do you expect this to change with 0.5? It’s been possible to write Lua plugins since 0.3, with luv becoming available in 0.4. I don’t see anything (besides the hype, maybe? :^) ) in 0.5 that would change the attitude of plugin authors towards Luarocks.
It’s true that we’ve had the ability to write Lua plugins for a while now, but there’s been a huge increase in the use of Lua during the later part of the 0.5 development cycle (the past year or so). This, to me, indicates that we may have passed an inflection point in the ease of using Lua for plugin development that will contribute to more people writing Lua plugins, and (potentially) more people using functionality from Luarocks libraries for those plugins (though I admit this also assumes that people need and know about the libraries available through Luarocks).
The hype is another factor, at least at first. And - noting that this runs dangerously close to self-promotion - I think that tools like
packer.nvim‘s Luarocks integration (which you can use even if you don’t usepackerfor plugin management) significantly lower the complexity cost of using Luarocks packages, while also requiring features from 0.5.I see. In my opinion there’s a fundamental issue with the with dependencies where as a plugin author you don’t want to rely on external dependencies (otherwise you’re inevitably going to end up with people opening issues after misconfiguring the dependencies for your plugin…) but vendoring is fairly unattractive too (messes up version control history, you have to manually update everything…). This issue was already there with Vimscript and I don’t think it can be solved by anything else than Vim/Neovim implementing a full-featured plugin manager (which I don’t think it should do).
Yeah, that’s definitely true too. I think this is easier with Luarocks because it already supports some basic versioning specification, so you can control what versions of dependencies your users get without needing to vendor. I definitely agree that there shouldn’t be a full-featured plugin manager baked into Vim/Neovim, though - people’s preferences and needs around plugin management are too disparate for there to be a good “one size fits all” option, in my opinion.
My point was that the package manager that downloads from luarocks itself becomes a dependency, which you want to avoid. So luarocks doesn’t really make things easier there :)
One way would be to use Python:
Or Ruby, Racket, Lua, Tcl, Perl, or even Visual Basic. It’s nice that Vim can interface with all that, but the big problem with that is that a lot of people don’t have the scripting support, and if you want things to work on Linux (all distros), macOS, and Windows is nigh-impossible. In reality It’s all pretty useless if as can never use it and have it work for everyone (the reason I didn’t try
:luais because my Vim isn’t compiled with it, and also, because I don’t really know Lua).Generally Vim tries to minimize required dependencies, which isn’t a bad thing. The Lua interpreter is pretty small so it could be integrated like libvterm or xdiff instead of linking to
liblua. But I guess it’s too late for that now :-/I think under-appreciated aspect is that Rust not only prevents safety issues from being exploitable, but in many cases prevents them from happening in the first place.
Contrast this with C where safety efforts are all focused on crashing as fast as possible. Things are nullable, but you get segfaults. Buffers are unsafe, but you have crashes on guard pages and canaries. Memory management is ad-hoc, but the allocator may detect an invalid
freeand abort.Rust in some places ensures safety by panicking too, but it also has plenty of tools that guarantee safety by construction, like move semantics, borrowing, iterators,
matchon options and results. So the bar is higher than merely “not exploitable because it crashes”. It’s not exploitable, because it works correctly.Interestingly, “crash early, crash often” is actually AFAIU an approach that can also lead to really robust software when properly embraced, as in Erlang. Personally I’m not convinced C is a particularly exemplary follower of this philosophy, at least as seen in the wild. With that said, I sometimes wonder if this isn’t something of a blind spot for Rust - i.e. stuff like cosmic rays (which I read start becoming noticeable at some scales), or processor bugs and attacks based on them. Maybe some of those risks can be mitigated with stuff like ECC memory? I’m certainly not an expert in this area FWIW, far from it. Just when thinking, I sometimes wonder if the Erlang philosophy is not the winner here in what it can achieve in the biggest horizon of robustness/resiliency (?)
Rust has the “crash early” philosophy too (asserts, indexing panic, overflows can panic), but it just doesn’t need it as often at run time. You wouldn’t want to crash just for the sake of crashing :)
Rust has all of C’s protections too. They’re added by LLVM, OS, and hardware.
Erlang’s “Let It Crash” philosophy is more than just ‘die a lot’ – the other part is that, recognizing that software can fail for a wide variety of reasons but that those reasons are frequently sporadic edge cases, you can get huge operational wins by having an intentional and nuanced system of recovery in-platform.
In practice what this means is that in Erlang/OTP, you frequently have tens, thousands, or millions of concurrent processes, as you might in an OS. Many of these processes are executing business logic, etc. But some of them are ‘supervisor’ processes, which detects when one of the processes it is monitoring has died, and then attempts to restart it (perhaps with a maximum number of attempts in a given time period). This is recursive; for complex applications you might have supervisor trees that are several layers deep.
The benefit here is that you start by writing the golden path. And then, sometimes, you just stop. Get illegal JSON input? Explode. Get an HTTP request you didn’t expect? Explode. Frequently, pragmatically, you don’t need to write the code that handles all of the garbage cases if your supervision tree is solid enough and your lightweight processes start ultra fast.
This philosophy has downsides – if the processes are carrying a large amount of important internal state, then sometimes them dying can be inconvenient because resurrection might involve an expensive rebuild step. So you need to pay more than the usual amount of attention to the stateful/stateless boundaries.
Rust at one point had this as a core idea, but abandoned it for, in my opinion, poor reasons. And now lands up grappling with janky concurrency coloring problems, as it unfortunately appears Zig will too.
The OTP monitoring system seems really really cool. However rust abandoned green threads for a very good reason: performance. When the goal is to be able to maximize performance, and compete with C++, you can’t exactly ask for heavy runtime with reduction counters and all the bells and whistles of BEAM, useful as they are.
The “coloring problem” caused by async stems from the same root causes: you can’t really have resizable stacks and lightweight threads in a language that is supposed to be able to run without even a heap allocator (“no-std” in rust parlance). That’s better left to higher-level languages like java or Go (or python if they had made less bad choices).
Here’s a more detailed explanation by Graydon Hoare, the original creator of Rust.
In short: Rust’s removal of green threads was for two reasons: 1) performance (as you mentioned), 2) difficult interoperation with C. Given Rust’s design constraints, including that you shouldn’t pay for what you don’t use, and that interoperation with C is expected to be easy, this approach didn’t work for Rust.
Edit: Replaced link to thread with a link to a collection, as Twitter’s default threading makes it easy to miss the second half.
Crash-early is fine, as long as the process has a supervisor process that promises to take care of restarting it. That’s the part that seems missing in a lot of systems (in C or otherwise.) Often it just seems kind of ad-hoc. Having implemented some IPC in C++ in the past using POSIX APIs, I remember it being nontrivial for the first process to handle the child’s death (but I may be misremembering.) Its certainly not something done for you, the way it is in Erlang.
If you are lucky. Note that this isn’t guaranteed and doesn’t always happen.
Yes, the worst part is when something overwrites some other (valid) memory, causing an error when that memory is next accessed (or worse no error at all). Pretty much your only recourse at that point is to whip out hardware watchpoints (which IME tends to be buggy and tank performance).
Open might return 0 and still be successful; the error check is sightly wrong.
Oh, imagine if Mozilla, Microsoft, and Google knew they can just fuzz their C code to get rid of all the bugs! These foolish companies have spent so much money and engineering time on their security programs, patching CVE after CVE, of which ~70% were caused by memory unsafety. If only they knew…
With C it’s always “all these vulnerabilities don’t count”, because you should have fuzzed more, you should have got better sanitizers, you should have got better programmers — for 40 years. The real futility is in “just don’t write bugs” approach.
The safer languages, on top of built-in safety guarantees, also support fuzzing and memory sanitizers, so you can perform your usual due diligence there too.
“Battle-tested” is a weak property. It only applies to software that’s unmaintained. As soon as you change the code, even a single line, that code isn’t proven to be reliable any more (remember when Debian broke SSH key generation to silence a compiler warning?). Being battle-tested is merely an effect of software getting shipped. So ship the new stuff, and it’ll get battle-tested too.
This is cryptographic code we’re talking about. That fuzzing test suite you need in C? You also need it in Rust. And now the only difference between C and Rust is the fact that with C, you need to run your existing test suite under Valgrind and the sanitizers. Push a button, wait for a minute at most, and you’ll get almost all the safety Rust guarantees.
Because unlike most application code, cryptographic code, thanks to being constant time, is stupidly easy to test. And test it you must, considering the stakes. It is one of the few cases where “just don’t write bugs” is both mandatory and possible.
I agree. What you really want is a kickass test suite.
Test suites are not enough. This is proven time and time again. Any class of bugs you can move from “has to be caught by a test or fuzzer” to “can’t occur in the first place” is a huge win. Cryptographic code should desire these properties even more because it is usually used when security and correctness is paramount.
I would love to go even further and be able to prove constant time algorithms or higher level mathematical relationships statically, etc. Perhaps with theorem provers.
The discussion of the tradeoffs should be around language ergonomics and platform support, not around “but c basically has the same guarantees if you squint”. It absolutely does not.
Cryptographic code really is different. Yes, you want correctness, safety and whatnot. More than for pretty much anything else, save real time avionics and lethal medical devices. Thing is, Rust does not help you there.
Oh but I am discussing tradeoffs there. C has (as of early 2020) substantially better platform support, significantly worse API, and in the specific case of cryptographic code, a negligible safety disadvantage.
Yes, for most domains, the safety advantage of Rust is freaking huge. I’m only talking about cryptographic code, that I have first hand experience with. Yes, you still need to have some kickass test suite, but Rust would need the exact same test suite anyway. Yes, you need to use sanitizers & Valgrind, but that’s often trivial for cryptographic code, since it tends to have zero dependencies to begin with.
I’m not squinting here, I’m just looking at the application domain at hand. In this particular case, C actually provides basically the same guarantees as Rust. I wouldn’t have expected this 4 years ago when I started working on Monocypher (I actually considered Rust, but favoured C’s portability), but now I can confidently take it for granted: libraries that:
are only marginally safer when implemented in Rust, provided they have a good test suite. Modern cryptographic code fits that bill. It’s probably the only niche that does.
That said, cryptographic code doesn’t stand in a vacuum. It must be used somehow. C APIs are naturally unsafe, and the rest of the application will likely hugely benefit from Rust’s guarantees. The crypto library must then have a Rust interface. We could write bindings, but it’s a significant hassle. I would never chose C over Rust for new cryptographic code for a Rust project. For old cryptographic code, I wouldn’t mind considering a reputable C library, though I’d likely favour a Rust library instead.
Would you trust a C crypto lib more than HACL*, a lib written in F* (a dependently-typed language) and formally verified for memory safety, correctness, and resistance to side-channel attacks?
I wouldn’t. Monocypher’s comparative advantage are its simplicity, it’s ease of use, its ease of deployment, and its portability (part of its portability comes from its relatively small code size). I did sacrifice confidence a bit to get there.
I respect your opinion but boy I don’t understand it. I would take a formally verified library any day, assuming it is an option. And I don’t get why someone in the field wouldn’t. Maybe some day I will become enlightened.
I don’t think Daniel Bernstein got a lot of flak when he failed to make NaCl and TweetNaCl formally verified. He’s Daniel Freaking Bernstein after all, world renowned cryptographer who successfully invented primitives. Or maybe because this was 2008, and formal verification wasn’t such a thing at the time?
I don’t think Frank Denis got a lot of flak when he failed to make Libsodium formally verified. After all, it started out as a repackaging of NaCl, itself not formally verified, and expanded since. Not long ago, security advocates were still chanting use Libsodium, use Libsodium, use Libsodium. On the supported platforms, it is indeed a very strong alternative, used by high profile projects and audited by one of the most reputable security companies out there.
Even I didn’t get a lot of flak for failing to make Monocypher formally verified, when I started out over 4 years ago. I did get a lot of flack for daring to write a crypto library without first building a reputation for myself (which is backwards: achievements build reputation, not the other way around). Though that was almost exclusively from people who didn’t even look at my work (as evidenced by their criticism).
Now there is no question that a formally verified crypto library maximises confidence. We can’t beat machine checked proofs. But we can approach them. Here’s how I did it:
Is that as good as a formal, machine checked proof of correctness of the generated binary code? Certainly not. But it damn sure approaches it, especially if your compiler is formally verified like ccert. So when I say I sacrificed confidence, that’s how little. While I would hesitate to trust it with the nuke launch codes, I would trust it with my payment card information.
Moreover a project using a crypto library must also consider other sources of errors. If the rest of the program isn’t formally verified, we’re kinda back to square one, where the whole thing must be thoroughly tested to make sure we don’t have some vulnerability or error lurking there. For instance, have we made sure that the network packets can’t be adversarially reordered without us noticing? Using authenticated encryption is not enough, we need message numbers or timestamps for this. So unless you formally prove the whole thing, that vaunted confidence will be significantly lowered, and the difference between a formally verified crypto library and a “merely” rigorously tested one is even slimmer.
As for the advantages, well: Monocypher is a single file library. One source file, one header file. One more of each if you really really need compatibility with Ed25519 (by default, Monocypher uses Blake2b instead of SHA-512 for signatures). Putting it in a C/C++ project is trivial: just copy & past the files in. Language bindings are easy to write, and people wrote several without me even needing to ask. The size of the binary is very small, and further increases Monocypher’s portability. Though it wasn’t the initial goal, it turns out Monocypher is very well suited to small 32-bit microcontrollers: doesn’t take much space, and it is fast.
Now I’m not saying a formally verified library can’t also be extremely portable and very easy to use. Strictly speaking, Monocypher itself could be formally verified (assuming the C standard can be at all formalised). But if it turns out the currently formally verified libraries are harder to deploy, or less well documented (Monocypher has exquisite documentation, thanks to a couple contributors), then the time you take learning how to use it is time you don’t take writing or testing your own program. By being so cheap to integrate, Monocypher can also contribute to the overall confidence of the projects that use it. Or at least lower their costs.
That being said, sometimes the cost of using formally verified cryptography is very low. Jason Donenfeld briefly used Monocypher in the Windows Wireguard installer (to check that the fetched binaries were the right ones), then at some point switched to formally verified code in very little time. As much as I’d like to deny it, this was the right move.
If those people come into a forum where I’m discussing the advantages of safe programming languages and formal verification and they argue what you argued, I would respond the same way. This isn’t a personal attack…
And all of the other text in your post, to me, further strengthens the case for safe languages (assuming ergonomics, portability, etc are on par, as we’ve discussed). Having lots of arbitrary application code to secure, having lots of programmers who would otherwise have to be extremely diligent with tooling, etc - it just isn’t going to happen.
I may have been more salty than I have any right to be, sorry about that. Rest assured, I perceived no such attack. The advantages of formally verified code are real, and I reckon I have a vested interest in ignoring them.
Speaking of which, a more recent post taught me that the first thing they wrote in Rust was the ASN.1 parsing. That kinda changes everything: the advantages of a safe language in this case is substantial. Chacha20 is easy to implement safely in C, but recursive parsing code is a whole ‘nother can of worms. I’ve written parsers, and C is never my first choice there. The compatibility problem sucks, but if they need ASN.1, that was probably the right move. (Modulo a slap on the wrist for failing to use semver.)
Incidentally, I believe safe languages have the potential to be even more portable than C currently is. Once you have a formal specification and a formally verified reference implementation, you can make guarantees far beyond C could ever provide. And if you find a way to compile to standard C itself (in a way that doesn’t rely on undefined/unspecified/implementation defined behaviour), then you instantly exceeds its portability, though talking to existing C code may still be a hassle.
I agree, even in the case of Monocypher: while I have personally been very diligent about its implementation, and can confidently claim it is solid, its interface is definitely unsafe. Because C, and because no dependency: it’s not just the buffer overflows, there’s also the lack of an RNG. Users can’t just generate a key pair, they have to provide the random private key themselves, and I can just pray they stumbled upon one of the many warnings in the manual about how they should use the OS’s RNG. Same problem with authenticated encryption: users have to provide the nonce themselves, and the onus is on them to avoid catastrophic nonce reuse.
Because of this, I strongly believe that a low level crypto library such as NaCl, Libsodium or Monocypher is not enough (they’re higher-level than the low-level operations of OpenSSL, but my work on protocols convinced me they’re still fairly low level). At the very least, I strongly encourage language bindings to provide a higher-level API where users no longer have to worry about providing clean random numbers.
I’m also working on higher-level protocols. Key exchange first, then I’ll tackle a file format. Those can still be free of I/O, small, simple, and in C (with the usual extreme required diligence, which I’ve become better at the last few years). The end game, I think, will be a high-level network library. I’ll have to tackle actual I/O this time, so extreme portability is not possible. Pure C is really not looking good with this one. A Rust or Zig implementation (both of which support async), with a C interface, are probably much better. I’m not sure yet, we’ll see.
I don’t think the original author of the article realizes that cryptographic code is different. His arguments might be right for cryptographic code, but wrong for code in general, which was what he intended. Probably a Gettier problem here.
Agreed, I also suspect he’s being right for the wrong reasons.
Author here.
You are probably right. I should go write some crypto code. (And not publish it, of course.)
The road to publishing your own crypto code is hard, but not impossible. If you see the need and have the motivation, there is a path.
I’ve actually read a lot of your articles already, for years. I’m a fan. :)
Sitting down to read that article now.
Author here.
I am aware of many reasons (though not all of course) where crypto code is different from
bc. However, what I do know is that crypto code should be small, thoroughly tested, and frozen. My argument in the post was “I did this amount of due diligence forbc, which is not ‘critical’ code, so have thecryptographyauthors done that much due diligence?”This seems like such a short-sighted, limited view. Software is not bug-free. And ruling out a class of bugs by choice of technology as a measure of improving overall robustness won’t fix everything, but at the very least it’s a trade-off that deserves more thorough analysis than this empty dismissal.
I think he is probably right (after all, he wrote it) when he says rewriting his bc in Rust would make it more buggy. I disagree this is an empty dismissal, since it is backed by his personal experience.
For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.
I think there’s a big difference between programs and libraries with stable requirements and those that evolve here. The bc utility is basically doing the same thing that it did 20 years ago. It has a spec defined by POSIX and a few extensions. There is little need to modify it other than to fix bugs. It occasionally gets new features, but they’re small incremental changes.
Any decision to rewrite a project is a trade off between the benefits from fixing the accumulated technical debt and the cost of doing and validating the rewrite. For something stable with little need of future changes, that trade is easy to see: the cost of the rewrite is high, the benefit is low. In terms of rewriting in a memory-safe language, there’s an additional trade between the cost of a memory safety vulnerability and the cost of the rewrite. The cost of Heartbleed in OpenSSL was phenomenal, significantly higher than the cost of rewriting the crypto library. In the cast of
bc, the cost of a memory safety bug is pretty negligible.Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs. The author talks about fuzzing, address sanitiser, and so on. These are great tools. They’re also completely unnecessary in a memory-safe language because they try to find classes of bugs that you cannot introduce in the first place in a memory-safe language (and they do so probabilistically, never guaranteeing that they’ve found them all).
If you’re starting a new project, then you need a really good reason to start it in C and pay the cost of all of that fuzzing.
There’s an implied assumption here that if a language is memory safe, those memory safe bugs will simply go away. In my experience, that is not quite true. Sometimes those memory safety bugs will turn into logic bugs.
Not to pick on Rust here, but in Rust it is very common to put values into an array and use array indices instead of pointers when you have some kind of self-referential data structure that’s impossible to express otherwise using rust’s move semantics. If you simply do such a naive transformation of your C algorithm, your code will be memory safe, but all your bugs, use after free, etc, will still be there. You just lifted them to logical bugs.
Rust has no good abstractions to deal with this problem, there are some attempts but they all have various practical problems.
Other languages like ATS and F* have abstractions to help with this problem directly, as well as other problems of logical soundness.
Right - but in lifting these from memory bugs to logic bugs, you get a runtime panic/abort instead of a jump to a (likely-attacker-controllable) address. That’s a very different kind of impact!
You don’t get a panic if you access the “wrong” array index. The index is still a valid index for the array. Its meaning (allocated slot, free slot, etc), is lost to the type system, though in a more advanced language it need not be. This later leads to data corruption, etc, just like in C.
It leads to a much safer variant of data corruption though. Instead of corrupting arbitrary memory in c or c++ (like a function pointer, vtable, or return address), you are only corrupting a single variable’s value in allocated and valid and aligned memory (like a single int).
You would get a panic in rust for every memory corruption bug that could cause arbitrary code execution, which is what matters.
Can you expand on this? I had expected the behaviour in Rust to be significantly safer than C here. In C, the data corruption caused by use-after free often allows an attacker to execute arbitrary code.
I totally see your point about logical corruption (including things like exposing critical secrets), but I don’t follow that all the way to “just like C”. How would an array index error be exploited in Rust to execute arbitrary code?
I have once written a bytecode interpreter in C++, for a garbage collected scripting language. I implemented my own two-space garbage collector. For performance reasons, I didn’t use
malloc()directly, but instead allocated a big enough byte array to host all my things. If I overflew that array, Valgrind could see it. But if I messed up it’s internal structure, no dice. That heap of mine was full of indices and sizes, and I made many mistakes that caused them to be corrupted, or somehow not quite right. And I had no way to tell.I solved this by writing my own custom heap analyser, that examined the byte array and tell me what’s in there. If I see all my “allocated” objects in order, all was well. Often, I would see something was amiss, and I could go and fix the bug. Had I written it in Rust instead, I would have had to write the exact same custom heap analyser. Because Rust wouldn’t have prevented me from putting the wrong values inside my array. It’s perfectly “safe” after all, to write gibberish in that array as long as I don’t overflow it.
Now could this particular bug lead to arbitrary code execution? Well, not quite. It would generate wrong results, but it would only execute what my C++/Rust program would normally execute. In this case however, I was implementing a freaking scripting language. The code an attacker could execute wasn’t quite arbitrary, but it came pretty damn close.
The effects of data corruption depend on what the code does with the data. This often means arbitrary code execution, but not always. It’s not a property of C, it’s a property of the code. This doesn’t change when you change the implementation language.
Fundamentally there is no semantic difference between a pointer in a C heap and an array index into a Rust array. In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code. It’s what the code does with the data that leads to exploitation (or not).
Of course Rust has many additional safety advantages compared to C, buffer overflows don’t smash the stack, etc, and using references in Rust if you can is safe. And when using references, there’s a great deal of correlation between Rust’s notion of memory safety and true logic safety. This is good! But many people don’t realise that this safety is predicated on the lack of aliasing. The borrow checker is only a mechanism to enforce this invariant, it’s not an operative abstraction. It’s the lack of aliasing that gets you the safety, not the borrow checker itself. When you give up aliasing, you lose a lot of what Rust can do for you. Virtually everybody understands that if you introduce unsafe pointers, they give up safety, but less people seem to understand that introducing aliasing via otherwise safe mechanism has the same effect. Of course, the program continues to be memory safe in Rust terms, but you lose the strong correlation between memory safety and logic safety that you used to have.
Not that there’s anything wrong with this, mind you, it’s just something people need to be aware of, just as they are already aware of the tradeoffs that they make when using unsafe. It does make a projection for the number of bugs that Rust can prevent in practice more difficult, though.
I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”. Even a rust alias logic bug dies not open up this kind of arbitrary code execution exposure because you can’t alias an int with a function pointer or a vtable or a return address on the stack, like you can in c or c++. You can only alias an int with an int in safe rust, which is an order of magnitude safer and really does eliminate an entire class of vulnerabilities.
In the security research world, we usually treat control of the program counter (the aptly named
ripon x86-64) as “arbitrary code execution.” You can do a surprising of programming using code that’s already in a process without sending any byte code of your own with return-oriented programming.But does Rust let you do that here? What does a snippet of Rust code look like that allows attacker-controlled indexing into an array escalate to controlling the program counter?
Surely you agree that “variables changing underfoot” implies “programs flow becomes different from what I expect”. That’s why we use variables, to hold the Turing machine state which influences the next state. A logical use after free means “variables changing underfoot”. You don’t expect a free array slot’s value (perhaps now reallocated) to change based on some remote code, but it does.
Right, but “program flow becomes different from what I expect, but it still must flow only to instruction sequences that the original program encoded” is much much safer than “program flow can be pointed at arbitrary memory, which might not even contain instructions, or might contain user-supplied data”.
With ROP, the program flow only goes through “instruction sequences that the original program encoded”, and yet ROP is pretty much fatal.
ROP is not possible when you index an array wrong in rust, what is your point?
And you can’t do rop in safe rust.
Maybe not directly within the native code of the program itself, but I think (at least part of) 4ad’s point is that that’s not the only level of abstraction that matters (the memory bug vs. logic bug distinction).
As an example, consider a CPU emulator written entirely in safe Rust that indexes into a u8 array to perform its emulated memory accesses. If you compile an unsafe program to whatever ISA you’re emulating and execute it on your emulator, a bad input could still lead to arbitrary code execution – it’s at the next semantic level up and not at the level of your program itself, but how much does that ultimately matter? (It’s not really terribly different than ROP – attacker-controlled inputs determining what parts of your program get executed.)
That’s admittedly a somewhat “extreme” case, but I don’t think the distinction between programs that do fall into that category and those that don’t is terribly clear. Nearly any program can, if you squint a bit, be viewed essentially as a specialized interpreter for the language of its config file (or command-line flags or whatever else).
There’s no distinction here. If your program implements a cpu emulator then your program can execute with no arbitrary code execution at all and still emulate arbitrary code execution on the virtual cpu. If you want the virtual program executing to not possibly execute arbitrary virtual instructions, you need to generate the virtual program’s instructions using a safe language too.
In most cases, though, arbitrary virtual code execution is less dangerous than arbitrary native code execution, though that’s beside the point.
So…we agree? My point was basically that attacker-controlled arbitrary code execution can happen at multiple semantic levels – in the emulator or in the emulated program (in my example), and writing the emulator in a safe language only protects against the former, while the latter can really be just as bad.
Though I realize now my example was poorly chosen, so a hopefully better one: even if both the emulator and the emulated program are written in memory-safe languages, if the emulator has a bug due to an array-index use-after-free that causes it to misbehave and incorrectly change the value of some byte of emulated memory, that destroys the safety guarantees of the emulated program and we’re back in arbitrary-badness-land.
Sure but this is just as meaningful as talking about a cpu hardware bug that might cause a native safe program to run amok. Technically true but not very useful when evaluating the safe programming language
Right, I agree, and safe rust aliasing that the GP described is not possible to control the program counter arbitrarily.
Yeah exactly, this is the part I thought @4ad was arguing was possible. Eg. in C, use-after-free often allows me to make the program start interpreting attacker-provided data as machine code. I thought this is what 4ad was saying was also possible in Rust, but I don’t think that’s what they are claiming now.
To me, that’s a big difference. Restricting the possible actions of a program to only those APIs and activities the original code includes, vs C where any machine code can be injected in this same scenario, is a major reduction in attack surface, to me.
One thing to note is that code is data and data is code, in a true, hard-mathematical sense.
The set of
and
is exactly the same (unbounded!). Of course it is much easier in practice to effect desired behavior when you can inject shell code into programs, but that’s hardly required. You don’t need to inject code with ROP either (of course ROP itself is not possible in Rust because of other mitigations, this is just an example).
Please note that in no way I am suggesting that Rust is doing anything bad here. Rust is raising the bar, which is great. I want the bar raised even higher, and we know for a fact that this is possible today both in theory and practice. Until we raise the bar, I want people to understand why we need to raise the bar.
At the end of a day you either are type safe or you aren’t. Of course the specifics of what happens when you aren’t type safe depend on the language!
PS: arrays can contain other things than integers, e.g. they can contain function pointers. Of course you can’t confuse an int with a function pointer, but using the wrong function pointer is pretty catastrophic.
I guess this is what I don’t understand, sorry for being dense. Can you show a concrete code example?
In my mind I see a program like this:
In my mind, there are two differences between this code in Rust and similar code in C:
This only allows the user to launch nuclear missiles; it does not allow them to, say, write to the harddrive or make network calls (unless one of the actions contained code that did that ofc); in C, I’d likely be able to make something like this call any system function I wanted to, whether machine code to do that was present in the original binary or not.
In Rust, this doesn’t allow arbitrary control flow, I can’t make this jump to any function in the binary, I can only trick it into running the wrong
Action; in C, I can callrunon any arbitrary object anywhere in the heap.ie. in C, this would let me execute anything in the binary, while in Rust it still has to abide by the control flow of the original program, I thought was the case, anyway.
I think you’re saying this is wrong, can you explain how/why and maybe show a code example if you can spare the time?
This is correct and 4ad is mistaken. I’m not sure why 4ad believes the two are equivalent; they aren’t.
“is exactly the same”
It simply isn’t, and I’m not sure why you think it is.
Do you have any links on this that you recommend?
Good analysis. You didn’t use the words, but this is a great description of the distinction between stocks and flows: https://en.wikipedia.org/wiki/Stock_and_flow. I wish more people talking about software paid attention to it.
Author here.
I would also argue that crypto should not change often, like
bc. You might add ciphers, or deprecate old ones, but once a cipher is written and tested, there should be very little need for it to change. In my opinion.Have they actually rewrote anything? Or have they instead selected a different crypto library they trust better than the previous one? On the one hand, Rust has no advantage over C in this particular context. On the other hand, they may have other reasons to trust the Rust library better than the C one. Maybe it’s better tested, or more widely used, or audited by more reputable companies.
If I take your word for it however, I have to disagree. Rewriting a cryptographic library in Rust is more likely to introduce new bugs, than it is to fix bugs that haven’t already been found and fixed in the C code. I do think however that the risk is slim, if they take care to also port the entire test suite as well.
In the Cryptography case isn’t the Rust addition some ASN.1 parsing code? This is cryptography adjacent but very much not the kind of different that your point about cryptography code makes. Parsing code unless it is very trivial and maybe not even then tends to be some of the more dangerous code you can write. In this particular case Rust is likely a large improvement in both ergonomics for the parsing as well as safety.
You’ve got a point. I can weaken it somewhat, but not entirely eliminate it.
I don’t consider ASN.1 “modern”. It’s over complicated for no good reason. Certificates can be much, much simpler than that: at each level, you have a public key, ID & expiration date, a certificate of the CA, and a signature from the CA. Just put them all in binary blobs, and the only thing left to parse are the ID & expiration date, which can be left to the application. And if the ID is an URL, and the expiration date is an 64-bit int representing seconds from epoch, there won’t be much parsing to do… Simply put, parsing certificate can be “very trivial”.
Another angle is that if you need ASN.1 certificates, then you are almost certainly using TLS, so you’re probably in a context where you can afford the reduced portability of a safer language. Do use the safer language in this case.
Yet another angle is that in practice, we can separate the parsing code from the rest of the cryptographic library. In my opinion, parsing of certificate formats do not belong to a low-level cryptographic library. In general, I believe the whole thing should be organised in tiers:
This is a fair point. When it comes down to it, whether I am right or wrong about it will only be seen in the consequences of the decision that they made.
Here’s the more thorough analysis you’re asking for: this is cryptographic code we’re talking about. Many assumptions that would be reasonable for application code simply does not apply here:
I wrote a crypto library, and the worst bug it ever had wasn’t caused by C, but by a logic error that would have happened even in Haskell. What little undefined behaviour it did have didn’t have any visible effect on the generated code.
Assuming you have a proper test suite (that tests all input & output lengths), and run that test suite with sanitisers & Valgrind, the kind of bug Rust fixes won’t occur in your cryptographic C code to begin with. There is therefore no practical advantage, in this particular case to using Rust over C. Especially when the target language is Python: you have to write bindings anyway, so you can’t really take advantage of Rust’s better APIs.
These bugs still occur in critical software frequently. It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion), and the safer language guarantees more than your suggestions would. And there’s also no risk of someone forgetting to run those things.
Yes they do. I was specifically talking about one particular kind of critical software: cryptographic code. It’s a very narrow niche.
In my 4 years of first hand experience writing cryptographic code, it’s really not. Rust needs the same test suite as C does, and turning on the sanitizers (or Valgrind) on this test suite is a command line away. The real advantage of Rust lies in its safer API (where you can give bounded buffers instead of raw pointers). Also, the rest of the application will almost certainly be much safer if it’s written in Rust instead of C.
Someone who might forget those things has no business writing cryptographic code at all yet, be it in C or in Rust. (Note: when I started out, I had no business writing cryptographic code either. It took over 6 months of people findings bugs and me learning to write a better test suite before I could reasonably say my code was “production worthy”.)
Rusts advantage goes much further than at the api boundary, but again the discussion should be around how to get safer languages more widely used (ergonomics, platform support) and not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”. To me it is like saying “super careful contractors with perfect tools don’t need safety gear”, except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?
Yes it does. In almost all domains. I’m talking about modern cryptographic code.
Write a spec. A formal one if possible. Then implement that spec for more platforms. Convincing projects to Rewrite It In Rust may work as a way to coerce people into supporting more platforms, but it also antagonises users who just get non-working software; such a strategy may not be optimal.
You’re not hearing me. I’m not talking in general, I’m talking about the specific case of cryptographic code (I know, I’m repeating myself.)
I really don’t. I root out potential mistakes by expanding my test suite as soon as I learn about a new class of bugs. And as it happens, I am painfully aware of the mistakes I made. One of them was even a critical vulnerability. And you know what? Rust wouldn’t have saved me.
Here are the bugs that Rust would have prevented:
#ifdef, though I don’t actually know.Here are the bugs that Rust would not have prevented:
Lesson learned: in this specific case, Rust would have prevented the unimportant bugs, and would have let the important ones slip through the cracks.
In this discussion, I think it is important to remind that cryptography developers are explicitly and intentionally not writing modern cryptographic code. One thing they want to use Rust on is ASN.1 parsing. Modern cryptographic practice is that you shouldn’t use ASN.1 and they are right. Implementing ASN.1 in Rust is also right.
So am I.
I disagree.
I disagree here too.
I partially agree. There is no such thing as a perfect test suite. A good crypto implementation should have a comprehensive test suite, of course, no matter the language. But that still isn’t as good as preventing these classes of bugs at compile time.
Not really the point. Regardless of how lucky or skilled you are that there are no known critical vulnerabilities in these categories in your code, that disregards both unknown vulnerabilities in your code, and vulnerabilities in other people’s code as well. A safe language catches all three and scales; your method catches only one and doesn’t scale.
Note that I did go the extra mile and went a bit further than Valgrind & the sanitisers. I also happen to run Monocypher’s test suite under the TIS interpreter, and more recently TIS-CI (from TrustInSoft). Those things guarantee that they’ll catch any and all undefined behaviour, and they found a couple bugs the sanitisers didn’t.
After that level of testing and a successful third party audit, I am confident there are none left.
There is no such code. I have zero dependencies. Not even the standard library. The only thing I have to fear now is a compiler bug.
I went out of my way not to scale. Yet another peculiarity of modern cryptographic code, is that I don’t have to scale.
Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project? Because it seemed like a broader statement originally.
I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.
I was talking about Monocypher specifically. Other projects do have dependencies, and any project that would use Monocypher almost certainly has dependencies, starting with system calls.
Fair enough. I was thinking from the project’s point of view: a given project only need one crypto library. A greenfield project can ditch backward compatibility and use a modern crypto library, which can be very small (or formally verified).
Yes, other people write cryptographic code. I myself added my own to this ever growing pile because I was unsatisfied with what we had (not even Libsodium was enough for me: too big, not easy to deploy). And the number of bugs in Monocypher + Libsodium is certainly higher than the number of bugs in Libsodium alone. No doubt about that.
Another reason why crypto libraries written in unsafe languages don’t scale, is the reputation game: it doesn’t matter how rigorously tested or verified my library is, if you don’t know it. And know it you cannot, unless you’re more knowledgeable than I am, and bother to audit my work yourself, which is prohibitively expensive. So in practice, you have to fall back to reputation and external signs: what other people say, the state of documentation, the security track record, issues from the bug tracker…
This made me twitch!
Why make a choice which prevents an entire class of bugs when you could simply put in extra time and effort to make sure you catch and fix them all?
Why lock your doors when you can simply stand guard in front of them all night with a baseball bat?
While personally would back the
cryptographydevs’ decision here, I think there is a legitimate discussion to be had around whether breaking compatibility for some long-standing users is the right thing to do. This post isn’t contributing well to that discussion.