The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
(dev lead of TypeScript here, hi!)
We definitely knew when choosing Go that there were going to be people questioning why we didn’t choose Rust. It’s a good question because Rust is an excellent language, and barring other constraints, is a strong first choice when writing new native code.
Portability (i.e. the ability to make a new codebase that is algorithmically similar to the current one) was always a key constraint here as we thought about how to do this. We tried tons of approaches to get to a representation that would have made that port approach tractable in Rust, but all of them either had unacceptable trade-offs (perf, ergonomics, etc.) or devolved in to “write your own GC”-style strategies. Some of them came close, but often required dropping into lots of unsafe code, and there just didn’t seem to be many combinations of primitives in Rust that allow for an ergonomic port of JavaScript code (which is pretty unsurprising when phrased that way - most languages don’t prioritize making it easy to port from JavaScript/TypeScript!).
In the end we had two options - do a complete from-scrach rewrite in Rust, which could take years and yield an incompatible version of TypeScript that no one could actually use, or just do a port in Go and get something usable in a year or so and have something that’s extremely compatible in terms of semantics and extremely competitive in terms of performance.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions). We still want a highly-separated API surface to keep our implementation options open, so Go’s interop shortcomings aren’t particularly relevant. Go has excellent code generation and excellent data representation, just like Rust. Go has excellent concurrency primitives, just like Rust. Single-core performance is within the margin of error. And while there might be a few performance wins to be had by using unsafe code in Go, we have gotten excellent performance and memory usage without using any unsafe primitives.
In our opinion, Rust succeeds wildly at its design goals, but “is straightforward to port to Rust from this particular JavaScript codebase” is very rationally not one of its design goals. It’s not one of Go’s either, but in our case given the way we’ve written the code so far, it does turn out to be pretty good at it.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions)
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Now that I’ve worked on a garbage collector, I see a sweet spot in languages like Go and C# – they have both value types deallocated on the stack and GC. Both Java and Python lack this semantic, so the GCs have to do more work, and the programmer has less control.
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a Foo* type as well as a Foo type, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.
I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between struct (value) and mutable struct (reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.
I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
$ go build -gcflags "-m" main.go
.\main.go:8:14: *y escapes to heap
.\main.go:11:13: x does not escape
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the node.parent reference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forward
Hejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with Object.freeze it isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have a node.parent reference.
So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the node.parent reference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.
The bad part is that then all the nodes you didn’t replace have chains of node.parent references that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutate node.parent, which means that 1) you can’t actually Object.freeze(node) and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.
See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the node.parent reference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.
Hm yeah it was a very good talk. My summary of the type checking part is
The input to the type checker is immutable ASTs
That is, parsing is “embarassingly parallel”, and done per file
They currently divide the program into 4 parts (e.g. 100 files turns into 4 groups of 25 files), and they do what I’d call “soft sharding”.
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess you have to de-duplicate the type errors? Because some type errors might come twice, since you are duplicating some work
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Under 8 cores and 8 threads, the parallel front end can reduce the clean build (cargo build with -Z threads=8 option) time by about 30% on average. (These crates are from compiler-benchmarks of rustc-perf)
I guess this is just because, empirically, you don’t get more than 3x speedup.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
Acknowledging some weak spots, Go’s in-proc JS interop story is not as good as some of its alternatives. We have upcoming plans to mitigate this, and are committed to offering a performant and ergonomic JS API.
I disagree with this, only because it’s imperialism. I’m British, in British English I write marshalling (with two of the letter l), sanitising (-sing instead of -zing except for words ending in a z), and -ise instead of -ize, among other things. You wouldn’t demand an Arabic developer to write all his comments in English for your sake for the sake of being idiomatic, would you?
I’ve worked for a few companies in Germany now, about half of them with their operating language being in German. All of them had comments being written exclusively in English. I don’t know how that is in other countries, but I get the impression from Europeans that this is pretty standard.
That said, my own preference is for American English for code (i.e. variable names, class names, etc), but British English for comments, commit messages, pull requests, etc. That’s because the names are part of the shared codebase and therefore standardised, but the comments and commit messages are specifically from me. As long as everyone can understand my British English, then I don’t think there’s much of a problem.
EDIT: That said, most of these suggestions feel more on the pedantic end of the spectrum as far as advice goes, and I would take some of this with a pinch of salt. In particular, when style suggestions like “I tend to write xyz” become “do this”, then I start to raise eyebrows at the usefulness of a particular style guide.
All of them had comments being written exclusively in English. I don’t know how that is in other countries, but I get the impression from Europeans that this is pretty standard.
Developers in China seem to prefer Chinese to English. When ECharts was first open-sourced by Baidu most of the inline comments (and the entire README) were in Chinese:
In Japan I feel like the tech industry is associated with English, and corporate codebases seem to use mostly English in documentation. However, many people’s personal projects have all the comments/docs in Japanese.
If someone wants to force everyone to spell something the same within a language they should make sure it’s spelled wrong in all varieties, like with HTTP’s ‘referer’.
The Go core developers feel so strongly about their speling that they’re wiling to change the names of constants from other APIs.
The gRPC protocol contains a status code enum (https://grpc.io/docs/guides/status-codes/), one of which is CANCELLED. Every gRPC library uses that spelling except for go-grpc, which spells it Canceled.
Idiosyncratic positions and an absolute refusal to concede to common practice is part and parcel of working with certain kinds of people.
We’re drifting off-topic, but I have to ask: gRPC is a Google product; Go is a Google product; and Google is a US company. How did gRPC end up with CANCELLED in the first place?!
You wouldn’t demand an Arabic developer to write all his comments in English for your sake for the sake of being idiomatic, would you?
If this is something other than a private pet project of a person who has no ambition of ever working with people outside of his country? Yes, yes I would.
I believe the advice is still applicable to non-native speakers. In all companies I worked for in France, developers write code in English, including comments, sometimes even internal docs. There are a lot of inconsistencies (typically mixing US English and GB English, sometimes in the same sentence.)
In my experience (LatAm) the problem with that is people tend to have pretty poor English writing skills. You end up with badly written comments and commit messages, full of grammatical errors. People were aware of this so they avoided writing long texts in order to limit their mistakes, so we had one-line PR descriptions, very sparse commenting, no docs to speak of, etc.
Once I had the policy changed for the native language (Portuguese) in PRs and docs they were more comfortable with it and documentation quality improved.
In Europe people are much more likely to have a strong English proficiency even as a second or third language. You have to know your audience, basically.
While I like to write paragraphs of explanation in-between code, my actual comments are rather ungrammatical, with a bit of git style verb-first, removing all articles and other things. Proper English feels wrong in these contexts. Some examples from my currently opened file:
; Hide map’s slider when page opens first time
;; Giv textbox data now
;;Norm longitude within -180-180
; No add marker when click controls
;; Try redundant desperate ideas to not bleed new markers through button
;; Scroll across date line #ToDo Make no tear in marker view (scroll West from Hawaii)
Those comments would most likely look weird to a person unfamiliar with your particular dialect.
In a small comment it’s fine to cut some corners, similar to titles in newspapers, but we can’t go overboard: the point of these things is to communicate, we don’t want to make it even more difficult for whoever is reading them. Proper grammar helps.
For clarification, this is not my dialect/way of speaking. But I see so many short interline comments like this, that I started thinking they feel more appropriate and make them too, now. Strange!
Is “hat” a standard term regularly used in the golang ecosystem for a specific thing and on the list given in the article? If not, it is not relevant to the point in the article.
(And even generalized: if it happens to be an important term for your code base or ecosystem, it probably makes sense to standardize on how to spell it. in whatever language and spelling you prefer. I’ve worked on mixed-language codebases, and it’d been helpful if people consistently used the German domain-specific terms instead of mixing them with various translation attempts. Especially if some participants don’t speak the language (well) and have to treat terms as partially opaque)
I had to solve this once. I maintain a library that converts between HTML/CSS color formats, and one of the formats is a name (and optional spec to say which set of names to draw from). HTML4, CSS2, and CSS2 only had “gray”, but CSS3 added “grey” as another spelling for the same color value, and also added a bunch of other new color names which each have a “gray” and a “grey” variant.
Which raises the question: if I give the library a hex code for one of these and ask it to convert to name, which name should it convert to?
The solution I went with was to always return the “gray” variant since that was the “original” spelling in earlier HTML and CSS specs:
I don’t think it’s really “imperialism”—firstly, “marshaling” isn’t even the preferred spelling in the US. Secondly in countries all over the world job listings stipulate English language skills all the time (even Arabic candidates) and the practice is widely accepted because facilitating communication is generally considered to be important. Lastly, while empires certainly have pushed language standardization as a means to stamp out identities, I don’t think it follows that all language standards exist to stamp out identities (particularly when they are optional, as in the case of this post).
“marshaling” isn’t even the preferred spelling in the US
What makes you say that? (Cards on the table, my immediate thought was “Yes, it is.” I had no data for that, but the ngram below suggests that the single l spelling is the (currently) preferred US spelling.)
I’ve come to the conclusion that I’ll not devote time to learning any other version control system, probably ever.
I just don’t have the same issues with git that some people seem to have. I learned the git model long ago and I can get out of any real trouble without really thinking about it. The command line experience for me is just muscle memory at this point, I don’t really care that the commands could be streamlined.
I gave Jujutsu a good once over, read @steveklabnik tutorial and everything, played with it for a bit, but after some time I was just left wondering “why am I doing this?”. It’d just take me much longer to learn JJ than any time it’d save me for the rest of my life.
Maybe I’ll be forced to eat my words one day. I applaud the idea of innovating in this space, however. It seems the newer generation has plenty of trouble with git, something that helps them is useful.
I think git’s probably the local optimum for the next 10-20 years, so I think you’re making a safe bet.
All these experiments suggest to me that people are feeling genuine friction with git though. I still think a “theory of patches” (perhaps related to the one from Darcs) could motivate a new version control system that permits new uses, enables users to get further with less work and fear, and maybe avoids most of the pitfalls. I don’t think we’re there yet though.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface. My own experience of git is like yours. It isn’t easy, but I’m used to it. The replacement will have to make a material improvement day-to-day, larger than a nicer interface. It is, today, really hard for me to imagine feeling the kind of leap that was subversion to git (or mercurial) emerging from a git-to-new-porcelain transition.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface.
As I’ve said here multiple times here and elsewhere, I’m a git fanboy. I personally don’t understand any of these efforts to revamp git’s UI. But I’m a grumpy old man who thinks craftspeople should learn to master their tools when these tools are ubiquitous and good enough. So don’t listen to me :) .
However, even though I agree that the git internals are amazing and well designed, I have to object to the sophism “These experiments also tell me that people agree that git has a very good internal model.”
With this type of logic, I could also state “The effort put into the Wine project tells me that people agree that Windows has a very good graphics API and very good binary format.”
I don’t think it’s the case. Whether it is for Wine or for jj and xit. I think all of these projects are targeting compatibility, and trying to prevent flamewars and bikeshedding over which VCS to use for specific project. I believe the idea behind using the git protocol is: you use xit, they use jj, I use git, everybody is happy.
Although I agree with your disagreement to the logic, I personally find Git’s object model to be rather intuitive. The only sound complaint I’ve encountered recently, imo, is that Git is unable to store directories, but as far as my understanding goes that could be represented as an empty tree object, if Git were to be patched to support that representation.
Notwithstanding that, I do believe that the reason jj/xit/got use the same plumbing as Git is about compatibility.
Also, it’s not necessarily an issue with the newer generation. Young-ish programmers that I know are generally comfortable with Git, including me, while it seems like people used to Mercurial/SVN/CVS who have a hard time groking it.
As I said, I’m a git fanboy. I do think that the git object model is great, but I also see issues. One of my biggest issue is that git is incapable to track renames and copies.
If I rename a file and largely modify that file in the same commit, git will think that I deleted a file called foo.txt, and committed a new different file (written from scratch) called bar.txt. If I want to merge modifications of foo.txt on the lines I didn’t touch, I will have merge conflicts.
Copy is another issue. If I copy foo.txt to bar.txt and modify bar.txt a little bit in one commit, the git object model will think that you just created a new file bar.txt from scratch. If you want to merge work that modified foo.txt before you copied it, the changes will not be propagated to bar.txt.
This is a very intentional decision by git, compared to other systems that tracked renames/copies with metadata (svn and hg, I think).
git is content-based, and the metadata is derived, rather than stored.
It is counterintuitive, but I think git’s choice is better, and I thought most VCS experts basically came around to that conclusion (?)
This rename and branch metadata issue was a big argument around 2007 when git was gaining popularity, but it seems to have been mostly settled …
I think one main problem is if you have a branch-heavy workflow, which turned out to be extremely useful in practice, then the most general solution is for renames to be derived from content. It just does a better job in practice.
Another under-rated thing is that git repositories are extremely backward compatible. A “branch” is almost nothing to the git model, just as a “rename” is literally nothing. This makes A LOT of things work more smoothly – the core is more elegant and compatible
If I want to merge modifications of foo.txt on the lines I didn’t touch, I will have merge conflicts.
I don’t personally run into that problem, and I rename all the time … I am not familiar with the details, but my guess is it’s because I commit very often
And also there are like a half a dozen merge strategies in git, and I guess the experts on big projects use those (but I don’t)
IME the alternatives always run into problems too … branches are cheap in git, but expensive in other systems – both in terms of time and cognitive overhead.
i.e. it doesn’t make sense to criticize the object model on theoretical grounds, when the systems that did the theoretically better thing actually behave worse in practice
I am not sure but Linus might have mentioned this at his ~2007 talk on git at Google, which was famously a bit abrasive (I was there):
And again, to change my mind, I would have to see a system that does better in practice … IME it’s a non-issue with git, and git does significantly better than its predecessors.
Most of the time using git mv instead of mv works for me although there are still a bunch of caveats. IMO this is a porcelain issue rather than an object model one
Pretty naturally, people who have used the feature of reliably-commit-attached branches have more issues with Git not allowing syncing the reflog than the people who were told no-real-branches is the only possibility and formed their model across this lack of functionality!
Yeah same here. To me, git just gets a lot of work done. It was a good 20%-50% permanent improvement over svn and hg in my mind, and that’s a huge win compounded over a decade.
I think many issues can be solved by simply committing more often, which I always did with every VCS. And the others issues I hit along ago and have workarounds for. Stack Overflow is invaluable here.
It’s funny to me that I don’t think I would be happy with git without Google/Stack Overflow, which is a sign of deficiency, but it’s still the best thing. (And yes I treat the command names and flags as opaque crap to memorize.)
I already spend too much time faffing with various tools, so for me git and vim are “set and forget”. On the other hand, I seem to have more problems with distros (Ubuntu was sort of ruined IMO, Debian is OK but can use improvement), and browsers (Firefox and Chromium both have issues).
Interesting, I used to think the same way. I considered myself a git power user and had no problems with it at all. I was very productive, comfortable with editing my history and could fix all my coworkers’ problems. Then jj clicked for me and I couldn’t go back.
I was still slightly stuck with git because it’s more performant in huge repos like the kernel. So I frantically searched for ways to get jj’s productivity and power in git. I finally found the config option rebase.rebaseMerges, which is the crucial missing piece to mimic the megamerge workflow of jj. Other config options are obviously essential too, but rebaseMerges is never really talked about online as far as I can tell.
It’s not at all perfect. git is still more manual work and more friction than jj. And some things still annoyingly don’t work at all. (Why can’t git do an octopus merge when there is a conflict?? jj can.)
So git can definitely be usable and productive. But if a power user like me, with a big config polished over years, cannot get the same power that jj gives you out-of-the boxes and with an intuitive CLI… I definitely see jj as the future “default” VCS people will be using and recommending.
While git is certainly cumbersome from time to time, I’ve learned to live with it. But it doesn’t stop to amaze me how many people just put up with GitHub. Pushing a branch, going to web, clicking a button, filling out a form? Why isn’t pushing a branch enough? And then reviewing online in a GUI so slow it takes a second to open a side-panel with the list of affected files, at least on large PRs. This all feels so backwards.
Aliases are a bad solution. Abbreviations in Fish have taken over 90% of my shell aliases, including even many of the git aliases I had. It’s just superior in every way.
It’s also better than the script in $PATH approach as they autocomplete and are more easily searchable in reverse history, plus you can template the command line and move the cursor where you want. Scripts or aliases can’t do that.
What I don’t use abbreviations for I use the script solution (rarely) or simple shell functions.
An abbreviation in fish expands after being typed, so you see the full command. There are some more advanced features such as templating, but I’m not experienced enough on that to speak too much about.
Thank you. You made me realize that that I need to use abbreviations way more often. I already do, but I still have some shell script in PATH that could just be abbreviations.
What I especially like about abbreviations is that when I copy&paste the outputs e.g. to document something or share instructions, my personal abbreviations are already expanded, so they don’t confuse other people.
The labeled-switch feature is a nice optimization for bytecode interpreters — it looks equivalent to the GCC extension for taking the address of a statement label.
This zig feature is somewhere between a switch in a loop and GNU C label addresses. The jump table remains implicit in the switch statement, but the dispatch happens at each continue. This eliminates the loop branches and makes each dispatch easier for the CPU’s branch predictor.
re-reading your comment on the Cpython tail calls, it sounds like this Zig feature is going to approximate “compute goto” in C, which is better than a switch statement, but still worse than the “tail calls with preserve_none” that Python is using. The computed goto is still really hard for the optimizer to reason about and allocate registers for. Is that correct?
That is so cool! It looks like it could have the same performance benefits without the need to define custom compiler intrinsic’s for calling conventions. I wonder if it will be as fast in practice.
No,. there’s a lot of policy discretion. The US government has access to any data stored in the US belonging to non-US persons without basic due process like search warrants. The data they choose to access is a policy question. The people being installed in US security agencies have strong connections to global far right movements.
In 2004 servers operated by Rackspace in the UK on behalf of Indymedia were handed over to the American authorities with no consideration of the legal situation in the jurisdiction where they were physically located.
/Any/ organisation- governmental or otherwise- that exposes themselves to that kind of risk needs to be put out of business.
I seem to remember an incident where instapaper went offline. The FBI raided a data centre and took a blade machine offline containing blade servers they had warrants for, and instapapers, which they didn’t. So accidents happen.
Yes, but in that case the server was in an American-owned datacenter physically located in America (Virginia), where it was within the jurisdiction of the FBI.
That is hardly the same as a server in an American-owned datacenter physically located in the UK, where it was not within the jurisdiction of the FBI.
Having worked for an American “multinational” I can see how that sort of thing can happen: a chain of managers unversed in the law assumes it is doing “the right thing”. Which makes it even more important that customers consider both the actual legal situation and the cost of that sort of foulup when choosing a datacenter.
The US government has access to any data stored in the US belonging to non-US persons without basic due process like search warrants.
Serious question, who’s putting data in us-west etc when there is eu data centres? And does that free rein over data extend to data in European data centres? I was under the impression that safe harbour regs protected it? But it’s been years since I had to know about this kind of stuff and it’s now foggy.
It does not matter where the data is stored. Using EU datacenters will help latency if that is where your users are, but it will not protect you from warrants. The author digs into this in this post, but unfortunately, it is in Dutch: https://berthub.eu/articles/posts/servers-in-de-eu-eigen-sleutels-helpt-het/
Serious question, who’s putting data in us-west etc when there is eu data centres?
A lot of non-EU companies. Seems like a weird question, not everyone is either US or EU. Almost every Latin American company I’ve worked for uses us-east/west, even if it has no US customers. It’s just way cheaper than LATAM data centers and has better latency than EU.
Obviously the world isn’t just US/EU, I appreciate that. This article is dealing with the trade agreements concerning EU/US data protection though so take my comment in that perspective.
I haven’t personally made up my mind on this, but one piece of evidence in the “it’s dramatically different (in a bad way)” side of things would be the usage of unvetted DOGE staffers with IRS data. That to me seems to indicate that the situation is worse than before.
Not sure what you mean—Operational Desert Storm and the Cold War weren’t initiated by the US nor were Iraq and the USSR allies in the sense that the US is allied with Western Europe, Canada, etc (yes, the US supported the USSR against Nazi Germany and Iraq against Islamist Iran, but everyone understood those alliances were temporary—the US didn’t enter into a mutual defense pact with Iraq or USSR, for example).
they absolutely 100% were initiated by the US. yes the existence of a mutual defense pact is notable, as is its continued existence despite the US “seeking to harm” its treaty partners. it sounds like our differing perceptions of whether the present moment is “dramatically different” come down to differences in historical understanding, the discussion of which would undoubtedly be pruned by pushcx.
This isn’t true, as the US has been the steward of the Internet and its administration has turned hostile towards US’s allies.
In truth, Europe already had a wake-up call with Snowden’s revelations, the US government spying on non-US citizens with impunity, by coercing private US companies to do it. And I remember the Obama administration claiming that “non-US citizens have no rights”.
But that was about privacy, whereas this time we’re talking about a far right administration that seems to be on a war path with US’s allies. The world today is not the same as it was 10 years ago.
hm, you have a good point. I was wondering why now it would be different but “privacy” has always been too vague a concept for most people to grasp/care about. But an unpredictable foreign government which is actively cutting ties with everyone and reneging on many of its promises with (former?) allies might be a bigger warning sign to companies and governments world wide.
I mean, nobody in their right mind would host stuff pertaining to EU citizens in, say, Russia or China.
Nil channels are useful because essentially any practical use of channels should have a select with multiple branches and nil lets you turn off one branch of the select while keeping the others working. Even the most basic thing you can do, send work to some worker goroutines and receive answered data back, will have two branches for send data and receive data and at a certain point you will have sent everything and want to turn the sending branch off while still receiving.
You don’t really need nil for that, even if it’s the easiest ATM.
The same thing can be done with an empty channel for select arms that receive, and a channel with a full buffer for send arms. The zero value for a channel could’ve instead been a valid channel with no buffer, which would satisfy both of the above.
I’m personally not a fan of having nil be a valid value for some blessed types, it’s pretty error prone IMO (as nil is in general).
Some of these things are historical patterns that have trickled down. Go really should be approached in the context of C and non-object oriented approaches to compound types. They were not trying to make an entirely new language with no baggage: they were comfortable with C and they just wanted something a bit better. If that’s a good thing or not is a different matter. I’d say for some things the defaults are not clear; some these things could be compile or runtime errors.
But much like in C a variable declaration does not initialize it (disregard zero values here); it’s also not a compile error to use an unitialized variable. Just like in C to both create and initialize things you resort to function patterns like makeX – make(X) in Go – or newX – NewX in Go. Just like in C passing an unitialized pointer to a struct as an argument to a function is allowed and the function decides what to do with it (e.g. passing a nil slice to append).
In the Plan9 source code you find these exact same patterns in C which makes the decisions in Go much clearer. For example the function that appends to a string automatically initializes it if the string pointer given to it is NULL. In the absense of a constructor-like approach this kind of late-initialization helps with the ergonomics when writing C. That’s why you also see the pattern if x == nil { x.init() } in some Go methods where x is a pointer receiver.
Having that frame of reference also help make some “weird” things clearer, like why you can append to a nil slice but not assign to a nil map – append() is a function and initializes the nil slice on the spot; somemap["key"] = "value" is a dereferencing operation and dereferencing a NULL pointer (the nil map) makes no sense. A whole bunch of other things also start to make more sense in that headspace.
I know why Go is how it is, and understand their decisions. Doesn’t mean I can’t disagree with some of them! “It is what it is” is not a good reason for status quo when designing, what at the time was, a new thing; especially when trying to make it fool/Googler proof ;)
Here specifically, the chan struct could’ve been designed so the zero value is an unbuffered empty channel, which makes sense for a zero value and avoids needing to add nil to the mix.
Yeah I wasn’t necessarily writing to you in particular but more in general for people that might pass this by and maybe get a better way to look at these issues. There’s plenty to disagree!
Digressing into the second topic, I think they were well justified in not straying from the status quo when creating Go. I believe many of the decisions they made were actually very good and showed foresight, even if the language does not cover all the bases. But that’s a longer conversation.
Your company needs to get rid of the silly junior / senior / staff distinction. It’s worse than the short-lived “rockstar” time, and I’m seriously disappointed that this disease is spreading world-wide. (Although given the zeitgeist, I’m half expecting full OR-1 to OF-10 soon)
The industry aligning on consistent, standardized levels across companies is one of the best things that we could achieve to improve software engineering as a professional discipline. I’m not asserting that the current zeitgeist is anywhere close to that, but it does at least seem to be moving in that way, especially in light of sites like https://levels.fyi and https://progression.fyi.
What is the actual problem that you have? What alternative do you propose?
In what way do you see this “improve software engineering as a professional discipline”?
If the consistency and standardization comes down to having a very low granularity of adjectives attached to job descriptions, what’s the gain? I can’t think of a single way where the sole knowledge that Bob Bobson is a “Senior SE” and joins my team would help right now.
You didn’t answer my questions and responded with non-sequiturs. Your response assumes that consistent leveling is not possible, which is fallacious. If you start from that position, it’s no surprise you feel that way.
Imagine that meaningful, standardized levels could be achieved, and please try to answer my questions. Consider it a fantasy if you wish. But being unwilling to consider the possibility means you’re just entrenched into your preconceived notion, in which case there’s no use in continuing the conversation.
Your first question was what my actual problem is. And the main one is that I don’t really see any problem this vague level naming does beyond giving HR departments some rough guidelines. Does that improve software engineering?
In my opinion, meaningful and standardized can’t be achieved here, in a way that helps one for actual products. Especially if it all boils down to a single rank/title. This reminds me a lot of ‘70s Dungeons & Dragons, where you’re called a “Lord” once your fighter reaches 9th level. And of course said warrior is equal proficient at wielding longswords as they’re awesome with glaive-guisarmes.
But in the real world, skills are not equally distributed. And to continue my fractal of stubbornness, I’m also not a fan of ranking those on a five star or 1 to 11 scale, as that’s literally too one-dimensional. I’ve worked with a lot of Java developers in the course of my career, and still would find it very hard to objectively rate someone’s grasp of “the language”. What could be meaningful for a project with a narrow scope, focused on umpteen existing APIs and a need for a quick on-ramp might not necessarily overlap with something where you start from a blank slate and a lot of design work and prototyping is needed. And yet another employer might need someone focused on a lot of nitty-gritty details because code reviews and security checks are a major part. Are these separate, sequential levels?
That’s what it boils down for me: Our industry isn’t uniform enough to provide a solid foundation on which you’d build a standard for measuring developer ability. And definitely not something on the level of junior/senior, white belt/black belt or tenderfoot/eagle.
First, thank you for the response. This is well considered and does answer my question.
So, what I interpret here is that your belief is that the “junior/senior/staff distinction is silly” because we don’t (yet) have a way to make the distinction useful to us. You may even believe that it’s entirely impossible to do so.
I think for me the difference is that I am considering the benefits that would be reaped if we could achieve consistent, standardized levels that have actual meaning. I don’t think that it’s simple, and to be honest, I highly doubt if anything close will be achieved within the next 10-20 years.
I can say that part of the difference in the way that you and I think is that I am less concerned with individual skills and more with impact, aptitude, etc. So, carrying along with your Java example for a moment, I don’t really care so much about an individual’s grasp of “Java the language” in the abstract, I care about their ability to gain the level of grasp that they need in any language (given sufficient time) to achieve their task at hand. I agree with you that a different depth of understanding is required for different tasks, but I also feel that depth of understanding for individual pieces of technology would be one of the easiest things to standardize on (with better certifications, no different from how we decide that civil engineers know enough to be granted a stamp with the power to impact hundreds of thousands of lives), but I also think that level of standardization isn’t particularly useful.
So, this is a lot of words to say, I mostly agree with a lot of your thoughts, but I would add a lot of “yets” and “untils”, because I don’t think it’s inevitable that things remain this way. Thinking about software engineering as more similar to actual engineering is the shift that I am imagining could be so impactful to our industry, but I also don’t think it’s something that is going to happen any time soon.
The industry aligning on consistent, standardized levels across companies is one of the best things that we could achieve to improve software engineering as a professional discipline.
The industry has no reliable way of grading people on those levels. One company’s junior is another company’s staff. The levels mean very little with no common base to agree on.
Plus it assumes a perfect level of meritocracy which doesn’t exist all around. I’ve personally seen very unskilled programmers — to the level where we had to fire them as the code was so subpar — rise to tech lead and then CTO in two years.
Abusing and subverting the levels doesn’t mean the levels are meaningless, it just means the meaning is being ignored. Saying someone was undeservedly promoted to tech lead and CTO inherently implies that the terms “tech lead” and “CTO” do have a proper meaning to you.
Even in a single company it depends on who the manager is and each person’s negotiation skill and drive.
I once participated in a technical interview for a senior dev (potential internal transfer) that couldn’t tell you the difference in complexity of searching for an element in an array vs a map. The worst part is I was asking those questions while not even senior myself at the time! It was a wakeup call that I really needed a raise and that the imposters are alive and well. This one came with a project director’s recommendation, the only reason we did the tech interview because on paper they were not a good fit either.
No reliable way yet, and your second paragraph is an indictment of exactly what happens when such standards do not exist, so you actually seem to be unintentionally arguing my point.
Something being difficult to achieve doesn’t mean we should throw up our hands and ignore it. Solving difficult problems is how we move forward.
This. I’ve worked both in companies with extremely rigid and well defined levels and those that were entirely flat. At least in my anecdotal experience the ones with well defined levels were far more pleasant to work in. The ones with a flat structure develop hard to articulate or combat “implicit levels” that are subject to cliques and relationships. Humans pretty naturally develop hierarchies, so I’ve found it’s better when they’re designed and not grown.
Doing this for specific content in most websites is a cat-and-mouse game that only leads to frustration, in my experience. After a few months the HTML layout or the CSS class names of these websites changes and the improvements are undone. This is specially true for the worst cases in Web usability: websites with tons of JavaScript where classes are generated and re-named every deploy.
After feeling the pain I gave up for many years; what I’ve found works today is to just use “reader mode”.
I do the same: reader mode works great and is only a click away which is peanuts since I’m already planning to spend time reading the page. It’s also available on websites I don’t intend to spend time on again and it often bypasses soft paywalls.
I use Stylus a bunch on websites that are decent but unusable on a large screen (4k 32”) because their content is all the way to the left which forces me to turn my head, uncomfortable if reading for a while.
Those websites are generally not changing their classes often. Think Sphinx based docs, LKML, man pages, etc.
The article is a bit all over the place with its arguments but I agree with the general principle.
What the article doesn’t seem to focus on is to me the main advantage: composability. Every command in the terminal can be roped together with a common interface. Some people belittle that interface as “stringly typed” but in comparison to an IDE or other graphical programs, where nothing is composable, it’s very good indeed.
Not really related to the topic, but one of the things that I most dislike in Go is the heavy usage of magic comments.
I understand the idea that they want new code to be parsable in older versions of the language, but considering how heavily the language relies on magic comments I can’t see why it didn’t had a preprocessor marker like C and # instead.
I wouldn’t stay it’s “heavy”, more like a sprinkling. There are only three magic comments that are generally used (build, embed, generate) and even then it’s sparingly. Grafana’s Tempo has ~170k lines of Go and only 5 of those lines are //go: directives. A very large class of programs will never use any of them.
It really depends, for example I have a small project of around 2k lines of code in Go and I had to use multiple build flags since the code is for a desktop application and I had to create different code paths for Linux/macOS/Windows.
It is even worse if you need to use CGo, since if you need to embed C code you basically need to write C code as comments. And since they’re comments, you don’t have syntax highlight at all (or at least I don’t know one text editor that supports it, and my Neovim does support embedded code since it works fine in Nix).
Now keep in mind that in general I don’t think it is a huge issue, but I still think the language would have been more elegant if it had something else to do those transformations. But again, this is Go, so maybe being ugly is better.
Quick reply, but it’s more about being simple rather than ugly. A preprocessor is even more magical and unwieldy than magic comments. C code often has endless runs of #ifdef ... #ifndef ... to separate build platforms making the code mostly unreadable. Not to mention the magic macros that change the code from underneath your feet so you don’t even know what you’re reading.
Even Rust’s macros allows one to create their own little language which no one else understands. Given how the express purpose of Go was to be easy to read for anyone not familiar with the codebase that’d have been a non-starter.
I figure the Go authors wanted, rightfully, to steer clear of the mess that is preprocessors.
Maybe I was not clear, I was not suggesting for Go to have a preprocessor, just that those kind of magic be indicated with a symbol different from comments. I suggested that it could be the marker of preprocessor in C (e.g. #), but not that this is a preprocessor per see.
The idea is because this creates a clean separation between what a comment is and what is something special interpreted by the compiler. Something that was different so it is easier to see that it is doing something, e.g. have different syntax highlight for it.
And the Linux kernel maintainers are just cool with this, huh? A disagreement over programming languages where the old guard refuses to give a single millimeter while folks are turning out high-quality, production code, leads to middle-school-caliber cliques, personal abuses, and predictably, maintainer burnout and resignations.
The Linux community is so much less with Marcan’s resignation, but it’s also so much less if nobody will rein in the LKML folks and maintainers, because this abuse will continue until Linux is an OS only for enterprise servers or things that can broadly look like them insofar as hardware support goes (see, Marcan’s notes about the death of the hobbyist hacker in the kernel and core low-level userspace). Linus - where you at, man?
I predict that if Linus steps in to issue a BDFL judgement, it will favor the LKML old guard.
Linux is already owned, for all practical purposes, by the LF member organizations, who are all big companies. Their common interest doesn’t favor technical innovation nearly so much as it does stability for existing business-critical application. So, yeah, enterprise servers, Android, and some of the embedded market.
Red Hat is funding the development of Nova, and graphics drivers are pretty much the first real world test for Rust in Linux. Microsoft employs several maintainers of the R4L core. Google pretty much was the first to look at Rust for a Binder replacement. All of them are also working on other projects adjacent to OSes in Rust (for example OpenVMM, Fuchsia, Coconuts Sam which was started by SUSE). So there’s a lot of desire for innovating with Rust among those companies.
Yes, Asahi GPU drivers are part of the “graphics drivers” category. :) Nova is the second within the category of graphics drivers.
Graphics drivers are also obviously dependent on the DMA functionality, meaning that Red Hat has an incentive to find a solution to the current issue and it will help Asahi as well.
My point wasn’t that LF members don’t want to use Rust in general (which is clearly false!) but that they may not have much shared incentive to advocate for Rust in the kernel. The R4L statements of support come from a much smaller set of orgs than the set of LF members. Funding or labor contributions aren’t as easy to find, but I’d be willing to bet we’re talking at least 2 orders of magnitude difference, and probably 3.
Though note that some of those companies supporting R4L are also some of the biggest contributors to Linux. The Linux Foundation itself doesn’t do much Linux development, and most LF members don’t do tons of Linux development. So a raw count doesn’t tell you much.
That would have been more true a couple of years ago, though. At the moment, those are the same groups coming under pressure to visibly start improving memory safety, and “we’re adopting Rust” is an easy signal to use.
At some point it’s easier to get memory safety features in standard C than to rewrite the Kernel significantly in Rust.
C23 didn’t standardize “attribute cleanup” but it’s not far fetched. It has memset_explicit which is a security-only feature.
As much as it is an important topic memory safety is clearly not a high priority for big players otherwise C wouldn’t be the most popular language for the past 50 years in OS development.
Rust makes a big deal out of it (and it is) but maintainers and companies are willing to trade all of it off for plain stability and compromise.
At some point it’s easier to get memory safety features in standard C than to rewrite the Kernel significantly in Rust.
Not really. The same problem exists here as with Rust - working with upstream. Grsecurity had open source patches for decades with mitigations that reduced vuln impact radically, across entire classes of attacks. LF could have paid Brad to upstream but they didn’t for political reasons, and when there have been more politically minded folks like Kees who have tried it’s taken decades just to cherry pick basics, and that’s been met with massive criticism and public insults from upstream as well.
The problem isn’t Rust, the problem isn’t C, the problem is and always will be upstream.
otherwise C wouldn’t be the most popular language for the past 50 years in OS development.
Massive survivorship bias etc. All major kernels have been in C, thus those interested in kernel dev tend to learn C, thus new kernels are written in C.
At some point it’s easier to get memory safety features in standard C than to rewrite the Kernel significantly in Rust.
So far, no one has managed to invent a good memory safety scheme that doesn’t require a lot of invasive changes to C or C++ code to work. Sean Baxter’s Safe C++ just added Rust’s borrow checking feature to C++, which ofc existing code structures don’t fit. I’m not aware of any proposal that would actually solve memory safety in C, just vague hand waving that it’s totally possible without too much effort.
At some point it’s easier to get memory safety features in standard C than to rewrite the Kernel significantly in Rust.
I’ve thought a lot about this. I hate how complicated Rust is and I would love there to be a simpler way to get what Rust offers. But all the proposals to make C more memory-safe are laughably weak in comparison, and when I think about what it would take to draw the rest of the owl, I end up with something pretty close to Rust.
In particular, we are not yet (and perhaps won’t ever be) able to prove the safety of real-world C (or any other sufficiently versatile memory-unsafe language) in general. So if we are to have any hope of solving this problem we need a way to encapsulate things whose safety can’t be automatically proved. I think it’s fair to say this implies generic programming capabilities on par with Rust’s.
Also, while bounds checking is somewhat feasible to retrofit to C if you’re willing to make it slower and break all ABI compatibility everywhere, memory management is not. There’s a reason nobody uses attribute cleanup: it’s just not very useful. Heck. C++ has just about every memory management aid short of a borrow checker, and none of that seems to be enough. Rust makes this problem tractable by restricting what you can do, which feels bad to me coming from C, but we can’t statically solve the general problem today, and the runtime option is basically garbage collection. Practically speaking you cannot apply Rust’s solution to C either because you’d have to rewrite enough C to totally defeat the point.
Maybe we’ll get garbage-collected, bounds-checked C (I guess that’s what CHERIoT is) before we get a critical mass of software written in Rust. I honestly have no idea how to call it. But I don’t think there is any change you could reasonably make to C to create a path of less resistance than “rewrite it in rust” or “magical memory-safe hardware”, and the not-memory-safe hardware that’s in use today will probably go on being used for some time.
Rust makes a big deal out of it (and it is) but maintainers and companies are willing to trade all of it off for plain stability and compromise.
I think you could say the same thing about safety (the other kind) improvements in many fields. People who make and sell products have, historically, needed to be threatened with quite big sticks to spend money on making the products not hurt people. At some point I think some stick-waving will take place in software too.
Heck. C++ has just about every memory management aid short of a borrow checker, and none of that seems to be enough
I’ve had the “privilege” of building and maintaining a now 6-year-old C++ application. Originally built as a prototype in early 2020, we chose C++17 because it was the newest standard that was supported by the embedded Linux distribution provided by our vendor. The application is very data-heavy and allocation heavy (soft real-time processing of 25MP images at 20fps). It has gone through a ton of evolution and has had modules written by… politely… people who aren’t C++ memory management experts.
While the number isn’t 0, we have had very very few bugs that were due to memory safety issues. The keyword new is not used explicitly anywhere in the codebase; std::shared_ptr and std::unique_ptr are used heavily. Arrays and vectors are very rarely accessed directly by index but rather using for (auto elem: vec) {...} or other iterator techniques. We don’t, currently, have any static analysis beyond compilation in our CI process; at one point we did use SonarQube and it did a good job of pointing out some questionable stuff in some of the early code.
You’re right, compared to Rust, our code has not been perfect from a memory-safety perspective. At the moment, through its current execution paths, it appears to be, at least according to valgrind. We’ve soak tested it for weeks on spare hardware and don’t seem to have any memory leaks. We’re doing thousands of allocations/sec with small objects, and enough large allocations every second to consume all 64GB of system RAM in about 2 minutes.
Might there be issues lurking somewhere? Maybe. From an engineering perspective though? This system works fantastic and if there are issues related to memory corruption they are incredibly rare in practice.
It’s worth observing (as more follow-up/color than any kind of “rebuttal”, since I don’t really disagree!) that broader standards like C23 are less relevant for projects like the Linux kernel. There was some early activity in the 2009 to 2013 era more based on clang analyzer & warnings, but it probably wasn’t until late 2013 about 11.5 years ago that they even regularly tested with clang. From 1990 until then (aka a quarter of a century), the Linux kernel was basically gcc-specific C. Any compiler that wanted to work with the Linux kernel was/is beholden to plenty of gcc-extensions like its inline asm syntax. So, the reality today is probably “compatibility with whatever extensions both clang & gcc support”, not any specific standard. (EDIT: and I make no claim as to “how close” to “good enough” various notions of “safety” one could realize this way. It’s more a point about C23 relevance vs. compilers willing to experiment more than a standards committee might be. A related observation might be that even kernel-specific C extensions for new static analysis like sparse have been considered fair game.)
As an example of pressure from one direction, the NSA report on memory safety from 2022 led eventually to a National Cybersecurity Strategy with a pillar about “shifting software liability” and USG investment in “memory-safe languages”. And this was repeated by all the Five Eye governments. That sort of thing implies it’ll be on the agendas of regulators, procurement officers, investors, and insurance companies, which puts it on the agenda for CIOs and CTOs.
We also have people like Mark Russinovich (Microsoft Azure CTO) saying C/C++ should not be used in new projects, and Microsoft does quite a bit with Linux nowadays.
The EU introduced product liability rules recently that may be a big redirection of priorities for embedded and appliance software folks.
Several governments, chiefly the US federal government. Also, while it doesn’t directly mention memory safety (I believe), the EU Cyber Resilience Act does mandate stricter cybersecurity standards and puts heavier liabilities on companies for faults in products with software elements. The oft-quoted figure that 70% of vulnerabilities are caused by memory safety issues is presumably not universal, but if it is true for Google and Microsoft products, then it seems like they have a lot of incentive now to make sure their products are memory safe.
The oft-quoted figure that 70% of vulnerabilities are caused by memory safety issues is presumably not universal, but if it is true for Google and Microsoft products, then it seems like they have a lot of incentive now to make sure their products are memory safe.
Android (cite): “Our data shows that issues like use-after-free, double-free, and heap buffer overflows generally constitute more than 65% of High & Critical security bugs in Chrome and Android.”
Android’s bluetooth and media components (cite): “Use-after-free (UAF), integer overflows, and out of bounds (OOB) reads/writes comprise 90% of vulnerabilities with OOB being the most common.”
iOS and macOS (cite): “Across the entirety of iOS 12 Apple has fixed 261 CVEs, 173 of which were memory unsafety. That’s 66.3% of all vulnerabilities.” and “Across the entirety of Mojave Apple has fixed 298 CVEs, 213 of which were memory unsafety. That’s 71.5% of all vulnerabilities.”
Chrome (cite): “The Chromium project finds that around 70% of our serious security bugs are memory safety problems.”
Microsoft (cite): “~70% of the vulnerabilities Microsoft assigns a CVE each year continue to be memory safety issues”
Firefox’s CSS subsystem (cite): “If we’d had a time machine and could have written this component in Rust from the start, 51 (73.9%) of these bugs would not have been possible.”
Ubuntu’s Linux kernel (cite): “65% of CVEs behind the last six months of Ubuntu security updates to the Linux kernel have been memory unsafety.”
That’s basically every OS and browser engine with significant market share. (If anyone has cited numbers on the BSDs, please let me know. I don’t have time to go digging myself.)
many of those projects also use tools like valgrind regularly in their builds from what I recall. It really looks like at some point or size in a codebase if it is C/C++ you are almost guaranteed to have memory safety vulnerabilities no matter what you do to prevent it.
I prefer to describe it as “Regardless of how capable of writing correct C or C++ individuals are, it’s been shown that people in groups cannot”.
It’s more diplomatic, given how emotionally invested in their own skills people can get.
…and more recently, I’ve started describing Rust as a lockout-tagout system for software development, based on the ways its designers chose to focus on maintainability of projects written in it. (eg. working to avoid the need to reason globally, providing a powerful type system for encoding invariants, etc.)
Honestly, Linux won’t be the dominant “new” server in a few decades and it’s exaaaactly because of the way upstream is supported. Linux will be relegated to “weird” hardware that IBM pays a fortune to maintain and provide enterprise support for.
I think the most important feature for me will be tracking tool dependencies with the go command directly. No more problems caused due to different tooling versions when collaborating with other people :)
I was lucky that most tools I am using for my projects can be used as libraries and can be used with my own little command line executable, which then can be used in a versioned manner. Here is an example where I’m using //go generate which wraps an internal command, which uses esbuild for building JavaScript and CSS assets.
Although I wrote a post, being very excited about go tool, this is still an unfortunate side effect of the pattern. Using a separate go.mod for them works, but has it’s own tradeoffs 🙃
That’s a possibility but you can always go back to what people were doing before with a tools package or doing go install @version on its own.
I haven’t run into an issue with the common dependency graph yet. The unfortunate thing about it is not being able to discern quickly your own dependencies from the tools’ in go.mod.
I’m confused as to why an HTML blog would strain a server more than a plain text blog. Serving this text as HTML might, POSSIBLY, double the size from 1kb to 2kb. Killing off the favicon would be just as effective, maybe moreso, for saving server resources.
Honestly, you’ve gone way overboard with the HTML there, too. You could drop the lang, the doctype, and the entire head block and it would work fine on 99.99% of browsers.
Anything less and the Nu HTML Checker emits a warning. Tidy output looks good for the above HTML, if we don’t mind the following warning about implicit tag:
Dropping the lang is bad for accessibility. Without the doctype it’s not valid standards mode HTML5. The charset is necessary if you have any non-ASCII characters. The viewport tag is needed for proper rendering on mobile.
Dropping the lang is bad for accessibility. Without the doctype it’s not valid standards mode HTML5. The charset is necessary if you have any non-ASCII characters. The viewport tag is needed for proper rendering on mobile.
I’m not making html5, just serving txt files.
I use BOM (byte order mask) in order to let the browser know that I’m using utf-8.
You can use the reading mode of your browser to display the text fine.
I don’t think txt files are so much of a problem in term of accessibility (I mean you can configure your screen reader to change the spoken language).
I had a terrible connection at home (where my small “server” is located), and I wanted to write in english (which is not my first language). I wanted to submit those articles on websites like here or hn, which can bring a lot of people over my bad connection, and I wanted to have pages as small as possible (I already got a kiss of death on my internet box by a sudden hn popularity, it’s not cool for the readers nor for me).
I argued with a friend that “you can start a blog within one minute”, and to prove I were right I executed and created the first txt file (I did this in a little bit over 10 minutes, I’m slow :/).
But I really like lightweight websites with html & css, and love to waste my time exploring funky lightweight designs from 1MB club (https://1mb.club/) :)
While I understand the spirit, you may be shooting yourself in the foot. If your intention is for people to read your articles, not just have the server serve them, you’d do well in improving the reader’s user experience just a bit. Plain text is just not great to read on desktop and on mobile it is unreadable (I mean literally unreadable because the font size is miniscule).
As it stands it’s a bit contradictory to care about readers not being able to see your website if it gets squeezed to death because when they are able they can’t read it anyway.
Holy crap, I had never heard about this “duel”. The way the author sort of debases Jeffries’ entire approach is something else:
Jeffries, however, does not believe in bigger pictures; his approach to software design is proudly myopic. He prevents himself from seeing the forest by pressing his face against the trees. And sometimes, as he moves from one tree to the next, he takes a moment to celebrate:
That’s harsh, but quality writing. Reminds me of Hunter S. Thompson’s Nixon eulogy. May I never be the target of this sort of breakdown.
A related but much kinder breakdown is McIlroy vs. Knuth, which is perhaps the opposite of this one (a critique of too much design).
Seems like a cool use, but I wonder: wouldn’t copying and pasting the schema SQL to an LLM prompt and asking it to write a query yield the same results? Or uploading the schema file as some LLMa allow. I don’t get why being sort of built in the DB makes it better.
I actually never thought of doing it but seeing this it seems like a great way to write pure SQL in more complex database schemas.
I agree with the points on code and project organization but I’d say these are more Rails/Ruby-oriented points and not a general issue. The main problem is that Ruby itself gives us very few tools to organize code: there’s no package-level segregation where one can think of packages like a separate unit from the rest, calling code from different modules is syntatically hard, plus the Rails community focus on arranging projects by taxonomy rather than subject (“models, views, tests” and not “authn, search, accounts”) means every folder is a grab-bag of different things that don’t belong together.
Different solutions have been tried over the years but they all depends on people following unwritten conventions. In all large Rails projects I’ve been on cross-cutting concerns show up everywhere in the codebase like a virus and it takes a lot of discipline to prevent it.
This isn’t a general problem that feels the same in every language though, it’s much less of a concern in Go and Rust for instance.
I am kind of principal engineer in the US company Hubstaff
How curious, I almost went to work for Hubstaff. I eventually declined the offer for a different place but it seemed like an interesting company. However I can’t even begin to imagine how mundane and unimportant software work must feel in the midst of a war…
I agree with the points on code and project organization but I’d say these are more Rails/Ruby-oriented points and not a general issue.
Well, yes and no :) Of course, it was a talk for a Ruby conference drawing from my mostly-Ruby experience (I probably should’ve included Ruby specificity in the title to post here). But I believe that the set of ideas/directions of thinking I am talking about (looking at code as text/stories, focusing on “truth” and “telling how it is”, attention to what “page of code” can tell, etc.)—even if sounding somewhat naive/idealistic, can be applied more universally. I’d probably choose other ways to illustrate it and build a narrative if targeting a more general audience. I’d actually probably be in the future :)
However I can’t even begin to imagine how mundane and unimportant software work must feel in the midst of a war…
Yep, pretty surreal at times! Not as much now (I am in relative safety, performing tasks not completely unlike my “civil” job, though much less mundane), but the first months in the army were quite wild. Reviewing code that handles things like “refactor the representation of a specific task tracking metric for small US companies” on a satellite connection in a heavily bombed frontline settlement between other duties… Fun times.
But the interesting thing is that you remain your old self even in those conditions. I started working again just a few weeks after the start of the full-scale invasion (when still a civilian/volunteer but in a city that was then very close to a frontline), and in a few months, I was able to be again interested in software development and start writing blog and trying stuff when I had time. Humans are weird like that.
the Rails community focus on arranging projects by taxonomy rather than subject (“models, views, tests” and not “authn, search, accounts”) means every folder is a grab-bag of different things that don’t belong together.
You’ll maybe laugh, but I did present Padrino as an alternative to Rails, for those that want different taxonomies at a conference and literally got shouted at by someone in the Q&A for what my problem with Rails is. I mean… the whole point of the talk was that there’s different tastes and Padrino caters to a different one.
Another issue I ran into was that I worked once at a company that did use Padrino and I did build a project setup that was similar to how you want it, using Padrinos mountable apps feature (Imagine a folder per concern, with an optional Rack stack in front of it). I got into a long discussion with their tech lead in which he was telling me I was not using the feature as intended. I’m the author of the mountable apps feature…
It’s sooooo hard to get these things out of peoples brains. Rails scaffolding was an amazing feature to show that webapps could be built swift. But it came with such an amount of damage.
I love Ruby, Rails was always… something I had to deal with.
One of the most important things I learned from school was from TA’ing a class, when I realized that different peoples’ brains can trace different paths through the same terrain. Go far enough and there’s seldom a single optimum path, even with the same starting and ending points.
Happy to. TBH, nowadays, check before you buy. Padrino is maintained and has a few gigantic deployments no one talks about (and I can’t), so it will stay around. But there’s also Hanami and all the others out there.
But I think it still strikes a super good balance between helping you, but also being able to pop open the hood and figure things out yourself. The whole project is is built against a stable and documented internal API that lived for a decade. So for example a newly generated project just generates you a boot.rb that you can actually follow and rearrange yourself. And because the API is stable, your re-arrangements will survive version upgrades (or… it’s a bug!).
It’s the project where I learned a ton about collaboration and I will always be thankful for that.
plus the Rails community focus on arranging projects by taxonomy rather than subject (“models, views, tests” and not “authn, search, accounts”) means every folder is a grab-bag of different things that don’t belong together.
Several large Ruby shops have created package tools like you describe. Shopify released theirs as packwerk. I haven’t used this one, but it looks very much like the one I have, perhaps due to employees cross-pollinating.
These are more generally called build-time (as opposed to run-time) dependencies. While tools are the most common use-case, another example of a build-time dependency would be a plugin for the build system. The main reason why we need to distinguish between the two types of dependencies is because we may need to build them with different configurations (for example, optimized for build-time and debug for run-time) or even for different targets in case of cross-compilation.
The biggest issue, in my opinion, is the shared version resolution. The main project and all tools must share a single version of each dependency.
I can see why all the tools would share a dependency but also forcing the run-time case to use the same version is strange. Simplicity of implementation seems like would be the only reason. Strictly speaking, even tools (but not build system plugins) need to share the same dependency since they each run in a separate process.
No, these are not build-time dependencies. When you run go build, these tools do not run. They are development-time dependencies. The programmer simply has these tools at their disposal to run for when they are developing code. It’s not a build-time thing.
Thanks for the clarification, I missed the part that the tools have to be run manually.
I feel conceptually development-time (I would call it development-only) is orthogonal to build/run-time: you can theoretically have a development-only run-time dependency (for example, some instrumentation support library which is only useful during development). So this Go tool thing would be more accurately called manually-run build-time dependency (which is necessarily development-only). It would still be built for host rather than target when cross-compiling, right?
I think this misses that the tool run time (and number of runs) vary drastically from build time and number of builds. Eg generating code for protocol, which might have happened once and then been rebuilt on hundreds of different commits. I’m not sure l calling this build time is all that useful.
What Go has is a special case of the following more general arrangement: a build-time dependency with its output stored in the repository (BTW, a practice which is frowned upon by many for valid reasons). In this setup the tool will only be executed (by the build system) when the input changes. Otherwise, the build will use the pre-generated output, even for a freshly cloned repository.
In the Go case, the “update output when input changes” part is handled manually, “for secure” I assume (or maybe “for simple”). But I still think it’s helpful to use more general terminology.
While I appreciate it I think your focus on terminology is making this more complicated than it has to be. go get -tool is essentially a package manager, nothing more than that. These tools might not be dependencies at all or might never be ran on any machine, development or otherwise.
A little further and we have macros! Though a lisper, I miss macros in Go when I’d like to precompute things, load data into the binary etc. As is, it sadly demands a separate prebuild step, computing and placing such information in .go files, which can then be compiled.
Does the embed package not work for that? Doesn’t involve a separate prebuild step and exposes the data quite nicely through the standard fs package. I’ve used that to embed, eg., templates and other external data.
I agree with a lot of this. I actually think the article is being too kind, in the wild I’ve seen that many Rails apps degrade into more and more CPU-bound, mainly due to coding decisions that they’re more or less expected to make (the Rails Way).
Most Rails performance issues are database issues
I’ve worked on some projects were rendering out the template was about an order of magnitude slower than the DB query itself.
I’d also argue a lot of the database issues are because Rails divorces you from the DB layer too much and favors more inefficient database design decisions (no materialized views, etc).
Most apps also use a background job runner, Sidekiq being the most popular, and background jobs often take care of lots of slow IO operations, such as sending e-mails, performing API calls, etc.
Sidekid ends up being the dumping ground for workloads that are too slow to do in-band on the request path. These include converting files, performing calculations, but also things that look IO-bound but have CPU-intensive work due to Ruby being slow such as loading thousands ActiveRecord objects in memory and iterating over them.
However, over the last couple of years, many people reported how YJIT reduced their application latency by 15 to 30%. Like Discourse seeing a 15.8-19.6% speedup with JIT 3.2, Lobsters seeing a 26% speedup, Basecamp and Hey seeing a 26% speedup or Shopify’s Storefront Renderer app seeing a 17% speedup.
Many of these companies actually take performance very seriously and were probably commiting fewer mistakes. I imagine the speedup would be even more significant in some companies without that know-how. That is to say the average large Rails app is in worse shape than Shopify’s.
I’ve worked on some projects were rendering out the template was about an order of magnitude slower than the DB query itself.
I’m curious about how you measured that. Several tools show view rendering time as slow but it was because the database fetching only hit during the template rendering (because Active Record is lazy)
I guess that part got confusing, I meant for that sentence to be read in conjunction with the first part about Rails leading one towards inefficiency. Yes, it was because N+1 queries and lazy loading popping up in the render phase. My point was that while those are problems in the database interaction the DB itself was not an issue, just the misuse of it which was mostly caused by following Rails recommendations. The ideal query eventually ran ~10x faster after a different schema design.
The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
Source: https://www.reddit.com/r/typescript/comments/1j8s467/comment/mh7ni9g
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
https://github.com/danos/yang/commit/c98b220f6a1da7eaffbefe464fd9e734da553af0
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
That is great explanation of “Why Go and not Rust?”
If you’re looking for “Why Go and not AOT-compiled C#?” see here: https://youtu.be/10qowKUW82U?t=1154s
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
That interview is really interesting, worth watching the whole thing.
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Related comment about GC and systems languages (e.g. once you have a million lines of C++, you probably want GC): https://lobste.rs/s/gpb0qh/garbage_collection_for_systems#c_rrypks
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
https://lobste.rs/s/2ah6bi/programming_without_pointers#c_5g2nat
[1] BTW Guy Steele’s famous 1998 “growing a language” actually advocated value types in Java. AFAIK as of 2025, “Project Valhalla” has not landed yet
Compilers written in OCaml are famous for being super-fast. See eg OCaml itself, Flow, Haxe, BuckleScript (now ReScript).
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Maybe, but given that Flow, its main competitor, distributes binaries for all major platforms: https://github.com/facebook/flow/releases/tag/v0.264.0
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Also Nim.
Also Julia.
There surely are others.
Yes good points, I left out Nim and Julia. And apparently Crystal - https://colinsblog.net/2023-03-09-values-and-references/
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a
Foo*type as well as aFootype, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between
struct(value) andmutable struct(reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
https://forum.golangbridge.org/t/how-gc-handles-interior-pointer/36195/5
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
Yes, that is a bit of control you give up with Go. Searching for “golang escape analysis”, this article is helpful:
https://medium.com/@trinad536/escape-analysis-in-golang-fc81b78f3550
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Yes, the stack/heap distinction is made by the compiler, not the programmer, in Go.
Why did you leave JS/TS off the list? They seem to have left it off too and that confuses me deeply because it also has everything they need
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the
node.parentreference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forwardHejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
Really, you should watch the interview https://youtu.be/10qowKUW82U
AIUI a typical LSP implementation integrates all the tools you listed so they are sharing a syntax tree already.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with
Object.freezeit isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have anode.parentreference.So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the
node.parentreference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.The bad part is that then all the nodes you didn’t replace have chains of
node.parentreferences that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutatenode.parent, which means that 1) you can’t actuallyObject.freeze(node)and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the
node.parentreference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.You are not joking, just the hack to make type checking itself parallel is well worth an entire hour!
Hm yeah it was a very good talk. My summary of the type checking part is
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
Can you share the timestamp to the discussion of this hack, for those who don’t have one hour?
I think this one: https://www.youtube.com/watch?v=10qowKUW82U&t=2522s
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
They also posted a (slightly?) different response on GitHub: https://github.com/microsoft/typescript-go/discussions/411
Yes please!
I disagree with this, only because it’s imperialism. I’m British, in British English I write marshalling (with two of the letter l), sanitising (-sing instead of -zing except for words ending in a z), and -ise instead of -ize, among other things. You wouldn’t demand an Arabic developer to write all his comments in English for your sake for the sake of being idiomatic, would you?
I’ve worked for a few companies in Germany now, about half of them with their operating language being in German. All of them had comments being written exclusively in English. I don’t know how that is in other countries, but I get the impression from Europeans that this is pretty standard.
That said, my own preference is for American English for code (i.e. variable names, class names, etc), but British English for comments, commit messages, pull requests, etc. That’s because the names are part of the shared codebase and therefore standardised, but the comments and commit messages are specifically from me. As long as everyone can understand my British English, then I don’t think there’s much of a problem.
EDIT: That said, most of these suggestions feel more on the pedantic end of the spectrum as far as advice goes, and I would take some of this with a pinch of salt. In particular, when style suggestions like “I tend to write xyz” become “do this”, then I start to raise eyebrows at the usefulness of a particular style guide.
Developers in China seem to prefer Chinese to English. When ECharts was first open-sourced by Baidu most of the inline comments (and the entire README) were in Chinese:
In Japan I feel like the tech industry is associated with English, and corporate codebases seem to use mostly English in documentation. However, many people’s personal projects have all the comments/docs in Japanese.
If someone wants to force everyone to spell something the same within a language they should make sure it’s spelled wrong in all varieties, like with HTTP’s ‘referer’.
The Go core developers feel so strongly about their speling that they’re wiling to change the names of constants from other APIs.
The gRPC protocol contains a status code enum (https://grpc.io/docs/guides/status-codes/), one of which is
CANCELLED. Every gRPC library uses that spelling except for go-grpc, which spells itCanceled.Idiosyncratic positions and an absolute refusal to concede to common practice is part and parcel of working with certain kinds of people.
We’re drifting off-topic, but I have to ask: gRPC is a Google product; Go is a Google product; and Google is a US company. How did gRPC end up with
CANCELLEDin the first place?!When you use a lot of staff on H-1B and E-3 visas, you get a lot of people who write in English rather than American!
Wait until you hear about the HTTP ‘Referer’ header. The HTTP folks have been refusing to conform to common practice for more than 30 years!
If this is something other than a private pet project of a person who has no ambition of ever working with people outside of his country? Yes, yes I would.
I believe the advice is still applicable to non-native speakers. In all companies I worked for in France, developers write code in English, including comments, sometimes even internal docs. There are a lot of inconsistencies (typically mixing US English and GB English, sometimes in the same sentence.)
In my experience (LatAm) the problem with that is people tend to have pretty poor English writing skills. You end up with badly written comments and commit messages, full of grammatical errors. People were aware of this so they avoided writing long texts in order to limit their mistakes, so we had one-line PR descriptions, very sparse commenting, no docs to speak of, etc.
Once I had the policy changed for the native language (Portuguese) in PRs and docs they were more comfortable with it and documentation quality improved.
In Europe people are much more likely to have a strong English proficiency even as a second or third language. You have to know your audience, basically.
While I like to write paragraphs of explanation in-between code, my actual comments are rather ungrammatical, with a bit of git style verb-first, removing all articles and other things. Proper English feels wrong in these contexts. Some examples from my currently opened file:
Those comments would most likely look weird to a person unfamiliar with your particular dialect.
In a small comment it’s fine to cut some corners, similar to titles in newspapers, but we can’t go overboard: the point of these things is to communicate, we don’t want to make it even more difficult for whoever is reading them. Proper grammar helps.
For clarification, this is not my dialect/way of speaking. But I see so many short interline comments like this, that I started thinking they feel more appropriate and make them too, now. Strange!
“If you use standard terms, spell them in a standard way” is not the same as “use only one language ever”.
Is “chapéu” or “hat” the standard way of spelling hat in Golang? If it’s “hat”, your standard is “only use American English ever”.
Is “hat” a standard term regularly used in the golang ecosystem for a specific thing and on the list given in the article? If not, it is not relevant to the point in the article.
(And even generalized: if it happens to be an important term for your code base or ecosystem, it probably makes sense to standardize on how to spell it. in whatever language and spelling you prefer. I’ve worked on mixed-language codebases, and it’d been helpful if people consistently used the German domain-specific terms instead of mixing them with various translation attempts. Especially if some participants don’t speak the language (well) and have to treat terms as partially opaque)
What? England had the word “hat” long before the USA existed.
I had to solve this once. I maintain a library that converts between HTML/CSS color formats, and one of the formats is a name (and optional spec to say which set of names to draw from). HTML4, CSS2, and CSS2 only had “gray”, but CSS3 added “grey” as another spelling for the same color value, and also added a bunch of other new color names which each have a “gray” and a “grey” variant.
Which raises the question: if I give the library a hex code for one of these and ask it to convert to name, which name should it convert to?
The solution I went with was to always return the “gray” variant since that was the “original” spelling in earlier HTML and CSS specs:
https://webcolors.readthedocs.io/en/latest/faq.html#why-does-webcolors-prefer-american-spellings
I thought you guys loved imperialism?
Imperialism is like kids, you like your own brand.
I don’t think it’s really “imperialism”—firstly, “marshaling” isn’t even the preferred spelling in the US. Secondly in countries all over the world job listings stipulate English language skills all the time (even Arabic candidates) and the practice is widely accepted because facilitating communication is generally considered to be important. Lastly, while empires certainly have pushed language standardization as a means to stamp out identities, I don’t think it follows that all language standards exist to stamp out identities (particularly when they are optional, as in the case of this post).
What makes you say that? (Cards on the table, my immediate thought was “Yes, it is.” I had no data for that, but the ngram below suggests that the single l spelling is the (currently) preferred US spelling.)
https://books.google.com/ngrams/graph?content=marshaling%2Cmarshalling&year_start=1800&year_end=2022&corpus=en-US&smoothing=3&case_insensitive=true
It’s imperialist to use social and technical pressure to “encourage” people to use American English so their own codebases are “idiomatic”.
I disagree. I don’t see how it is imperialism in any meaningful sense. Also “pressure” is fairly absurd here.
I’ve come to the conclusion that I’ll not devote time to learning any other version control system, probably ever.
I just don’t have the same issues with git that some people seem to have. I learned the git model long ago and I can get out of any real trouble without really thinking about it. The command line experience for me is just muscle memory at this point, I don’t really care that the commands could be streamlined.
I gave Jujutsu a good once over, read @steveklabnik tutorial and everything, played with it for a bit, but after some time I was just left wondering “why am I doing this?”. It’d just take me much longer to learn JJ than any time it’d save me for the rest of my life.
Maybe I’ll be forced to eat my words one day. I applaud the idea of innovating in this space, however. It seems the newer generation has plenty of trouble with git, something that helps them is useful.
I think git’s probably the local optimum for the next 10-20 years, so I think you’re making a safe bet.
All these experiments suggest to me that people are feeling genuine friction with git though. I still think a “theory of patches” (perhaps related to the one from Darcs) could motivate a new version control system that permits new uses, enables users to get further with less work and fear, and maybe avoids most of the pitfalls. I don’t think we’re there yet though.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface. My own experience of git is like yours. It isn’t easy, but I’m used to it. The replacement will have to make a material improvement day-to-day, larger than a nicer interface. It is, today, really hard for me to imagine feeling the kind of leap that was subversion to git (or mercurial) emerging from a git-to-new-porcelain transition.
As I’ve said here multiple times here and elsewhere, I’m a git fanboy. I personally don’t understand any of these efforts to revamp git’s UI. But I’m a grumpy old man who thinks craftspeople should learn to master their tools when these tools are ubiquitous and good enough. So don’t listen to me :) .
However, even though I agree that the git internals are amazing and well designed, I have to object to the sophism “These experiments also tell me that people agree that git has a very good internal model.”
With this type of logic, I could also state “The effort put into the Wine project tells me that people agree that Windows has a very good graphics API and very good binary format.”
I don’t think it’s the case. Whether it is for Wine or for jj and xit. I think all of these projects are targeting compatibility, and trying to prevent flamewars and bikeshedding over which VCS to use for specific project. I believe the idea behind using the git protocol is: you use xit, they use jj, I use git, everybody is happy.
Although I agree with your disagreement to the logic, I personally find Git’s object model to be rather intuitive. The only sound complaint I’ve encountered recently, imo, is that Git is unable to store directories, but as far as my understanding goes that could be represented as an empty tree object, if Git were to be patched to support that representation.
Notwithstanding that, I do believe that the reason jj/xit/got use the same plumbing as Git is about compatibility.
Also, it’s not necessarily an issue with the newer generation. Young-ish programmers that I know are generally comfortable with Git, including me, while it seems like people used to Mercurial/SVN/CVS who have a hard time groking it.
As I said, I’m a git fanboy. I do think that the git object model is great, but I also see issues. One of my biggest issue is that git is incapable to track renames and copies.
If I rename a file and largely modify that file in the same commit, git will think that I deleted a file called
foo.txt, and committed a new different file (written from scratch) calledbar.txt. If I want to merge modifications offoo.txton the lines I didn’t touch, I will have merge conflicts.Copy is another issue. If I copy
foo.txttobar.txtand modifybar.txta little bit in one commit, the git object model will think that you just created a new filebar.txtfrom scratch. If you want to merge work that modifiedfoo.txtbefore you copied it, the changes will not be propagated tobar.txt.This is a very intentional decision by git, compared to other systems that tracked renames/copies with metadata (svn and hg, I think).
git is content-based, and the metadata is derived, rather than stored.
It is counterintuitive, but I think git’s choice is better, and I thought most VCS experts basically came around to that conclusion (?)
This rename and branch metadata issue was a big argument around 2007 when git was gaining popularity, but it seems to have been mostly settled …
I think one main problem is if you have a branch-heavy workflow, which turned out to be extremely useful in practice, then the most general solution is for renames to be derived from content. It just does a better job in practice.
Another under-rated thing is that git repositories are extremely backward compatible. A “branch” is almost nothing to the git model, just as a “rename” is literally nothing. This makes A LOT of things work more smoothly – the core is more elegant and compatible
I don’t personally run into that problem, and I rename all the time … I am not familiar with the details, but my guess is it’s because I commit very often
And also there are like a half a dozen merge strategies in git, and I guess the experts on big projects use those (but I don’t)
IME the alternatives always run into problems too … branches are cheap in git, but expensive in other systems – both in terms of time and cognitive overhead.
i.e. it doesn’t make sense to criticize the object model on theoretical grounds, when the systems that did the theoretically better thing actually behave worse in practice
I am not sure but Linus might have mentioned this at his ~2007 talk on git at Google, which was famously a bit abrasive (I was there):
https://www.youtube.com/watch?v=idLyobOhtO4
I think it’s one of those things where he had the contrary opinion at the time, was abrasive about it, and turned out to be right
Here is some rant I googled – I didn’t read the whole thing – but it is obviously extremely intentional and purposeful
https://gist.github.com/borekb/3a548596ffd27ad6d948854751756a08
And again, to change my mind, I would have to see a system that does better in practice … IME it’s a non-issue with git, and git does significantly better than its predecessors.
Most of the time using
git mvinstead ofmvworks for me although there are still a bunch of caveats. IMO this is a porcelain issue rather than an object model onePretty naturally, people who have used the feature of reliably-commit-attached branches have more issues with Git not allowing syncing the reflog than the people who were told no-real-branches is the only possibility and formed their model across this lack of functionality!
Yeah same here. To me, git just gets a lot of work done. It was a good 20%-50% permanent improvement over svn and hg in my mind, and that’s a huge win compounded over a decade.
I think many issues can be solved by simply committing more often, which I always did with every VCS. And the others issues I hit along ago and have workarounds for. Stack Overflow is invaluable here.
It’s funny to me that I don’t think I would be happy with git without Google/Stack Overflow, which is a sign of deficiency, but it’s still the best thing. (And yes I treat the command names and flags as opaque crap to memorize.)
I already spend too much time faffing with various tools, so for me git and vim are “set and forget”. On the other hand, I seem to have more problems with distros (Ubuntu was sort of ruined IMO, Debian is OK but can use improvement), and browsers (Firefox and Chromium both have issues).
Interesting, I used to think the same way. I considered myself a git power user and had no problems with it at all. I was very productive, comfortable with editing my history and could fix all my coworkers’ problems. Then jj clicked for me and I couldn’t go back.
I was still slightly stuck with git because it’s more performant in huge repos like the kernel. So I frantically searched for ways to get jj’s productivity and power in git. I finally found the config option
rebase.rebaseMerges, which is the crucial missing piece to mimic the megamerge workflow of jj. Other config options are obviously essential too, butrebaseMergesis never really talked about online as far as I can tell.It’s not at all perfect. git is still more manual work and more friction than jj. And some things still annoyingly don’t work at all. (Why can’t git do an octopus merge when there is a conflict?? jj can.)
So git can definitely be usable and productive. But if a power user like me, with a big config polished over years, cannot get the same power that jj gives you out-of-the boxes and with an intuitive CLI… I definitely see jj as the future “default” VCS people will be using and recommending.
While git is certainly cumbersome from time to time, I’ve learned to live with it. But it doesn’t stop to amaze me how many people just put up with GitHub. Pushing a branch, going to web, clicking a button, filling out a form? Why isn’t pushing a branch enough? And then reviewing online in a GUI so slow it takes a second to open a side-panel with the list of affected files, at least on large PRs. This all feels so backwards.
It’s possible to check out GitHub PRs locally…
Except that if you merge that ref (or the actual source ref) directly with Git, GitHub won’t detect the PR as being merged
What a headache
I have to agree, I have no problems with git.
All these alternative systems seems to come from people who want simpler tools and do not understand git.
Do you really think authors of these alternative systems do not understand git? People like @martinvonz?
Aliases are a bad solution. Abbreviations in Fish have taken over 90% of my shell aliases, including even many of the git aliases I had. It’s just superior in every way.
It’s also better than the script in $PATH approach as they autocomplete and are more easily searchable in reverse history, plus you can template the command line and move the cursor where you want. Scripts or aliases can’t do that.
What I don’t use abbreviations for I use the script solution (rarely) or simple shell functions.
What’s the difference, and what are the alternatives for those who don’t use fish?
An abbreviation in fish expands after being typed, so you see the full command. There are some more advanced features such as templating, but I’m not experienced enough on that to speak too much about.
That’s super cool! I have been using fish for a couple of years now and haven’t come across abbreviations yet – looks like it is time to give it a go!
As far as I know there are no alternatives in other shells. It’s something I’ve seen only in Fish.
The doc has some examples of its usefulness: https://fishshell.com/docs/current/cmds/abbr.html
You can set ZSH to tab expand aliases, and you can expand them with a keybind in bash as well. https://superuser.com/questions/1514569/how-to-expand-aliases-inline-in-zsh
I see. In comparison, I don’t really see why you’d say that “aliases are a bad solution”, then? Looks mostly like a matter of taste to me.
Thank you. You made me realize that that I need to use abbreviations way more often. I already do, but I still have some shell script in PATH that could just be abbreviations.
What I especially like about abbreviations is that when I copy&paste the outputs e.g. to document something or share instructions, my personal abbreviations are already expanded, so they don’t confuse other people.
Why does your shell not autocomplete$PATHbinary names?Never mind, I realised you meant it breaks the autocomplete of the binary’s arguments/flags/etc.
The labeled-switch feature is a nice optimization for bytecode interpreters — it looks equivalent to the GCC extension for taking the address of a statement label.
Tokenizers/parsers too. They already converted their tokenizer to use labeled switches.
How would this be an optimization for bytecode interpreters? It looks like it is just syntax sugar
See https://lobste.rs/s/ui5fzs/cpython_tail_call_interpreter_merged_for#c_c5fwac
This zig feature is somewhere between a switch in a loop and GNU C label addresses. The jump table remains implicit in the switch statement, but the dispatch happens at each continue. This eliminates the loop branches and makes each dispatch easier for the CPU’s branch predictor.
re-reading your comment on the Cpython tail calls, it sounds like this Zig feature is going to approximate “compute goto” in C, which is better than a switch statement, but still worse than the “tail calls with preserve_none” that Python is using. The computed goto is still really hard for the optimizer to reason about and allocate registers for. Is that correct?
That’s my understanding, yes.
That is so cool! It looks like it could have the same performance benefits without the need to define custom compiler intrinsic’s for calling conventions. I wonder if it will be as fast in practice.
The post described it. Jumping from one case to the next saves some instructions and helps branch prediction. It’s a well known technique in C.
It’s just as safe as it’s always been.
No,. there’s a lot of policy discretion. The US government has access to any data stored in the US belonging to non-US persons without basic due process like search warrants. The data they choose to access is a policy question. The people being installed in US security agencies have strong connections to global far right movements.
In 2004 servers operated by Rackspace in the UK on behalf of Indymedia were handed over to the American authorities with no consideration of the legal situation in the jurisdiction where they were physically located.
/Any/ organisation- governmental or otherwise- that exposes themselves to that kind of risk needs to be put out of business.
I seem to remember an incident where instapaper went offline. The FBI raided a data centre and took a blade machine offline containing blade servers they had warrants for, and instapapers, which they didn’t. So accidents happen.
Link: https://blog.instapaper.com/post/6830514157
Yes, but in that case the server was in an American-owned datacenter physically located in America (Virginia), where it was within the jurisdiction of the FBI.
That is hardly the same as a server in an American-owned datacenter physically located in the UK, where it was not within the jurisdiction of the FBI.
Having worked for an American “multinational” I can see how that sort of thing can happen: a chain of managers unversed in the law assumes it is doing “the right thing”. Which makes it even more important that customers consider both the actual legal situation and the cost of that sort of foulup when choosing a datacenter.
The FBI has offices around the world.
https://www.fbi.gov/contact-us/international-offices
Serious question, who’s putting data in
us-westetc when there is eu data centres? And does that free rein over data extend to data in European data centres? I was under the impression that safe harbour regs protected it? But it’s been years since I had to know about this kind of stuff and it’s now foggy.It does not matter where the data is stored. Using EU datacenters will help latency if that is where your users are, but it will not protect you from warrants. The author digs into this in this post, but unfortunately, it is in Dutch: https://berthub.eu/articles/posts/servers-in-de-eu-eigen-sleutels-helpt-het/
I re-read the English article a bit better and see he addresses it with sources and linked articles. Saturday morning, what can I say.
A lot of non-EU companies. Seems like a weird question, not everyone is either US or EU. Almost every Latin American company I’ve worked for uses us-east/west, even if it has no US customers. It’s just way cheaper than LATAM data centers and has better latency than EU.
Obviously the world isn’t just US/EU, I appreciate that. This article is dealing with the trade agreements concerning EU/US data protection though so take my comment in that perspective.
I don’t see how this is at odds with the parent comment?
That is the one good thing. It has always been unsafe, but now people are finally starting to understand that.
Because it’s dramatically less safe. Everyone saying “it’s the same as before” has no clue what is happening in the US government right now.
And everyone saying it’s dramatically different has no clue what has happened in the US government in the past.
I haven’t personally made up my mind on this, but one piece of evidence in the “it’s dramatically different (in a bad way)” side of things would be the usage of unvetted DOGE staffers with IRS data. That to me seems to indicate that the situation is worse than before.
yeah could be
You’re incorrect. The US has never had a government that openly seeks to harm its own allies.
What do you mean? Take Operation Desert Storm. Or the early Cold War.
Not sure what you mean—Operational Desert Storm and the Cold War weren’t initiated by the US nor were Iraq and the USSR allies in the sense that the US is allied with Western Europe, Canada, etc (yes, the US supported the USSR against Nazi Germany and Iraq against Islamist Iran, but everyone understood those alliances were temporary—the US didn’t enter into a mutual defense pact with Iraq or USSR, for example).
they absolutely 100% were initiated by the US. yes the existence of a mutual defense pact is notable, as is its continued existence despite the US “seeking to harm” its treaty partners. it sounds like our differing perceptions of whether the present moment is “dramatically different” come down to differences in historical understanding, the discussion of which would undoubtedly be pruned by pushcx.
My gut feeling says that you’re right, but actually I think practically nobody knows whether you are or not. To take one example, it’s not clear whether the US government is going to crash its own banking system: https://www.crisesnotes.com/how-can-we-know-if-government-payments-stop-an-exploratory-analysis-of-banking-system-warning-signs/ . The US governmant has done plenty of things that BAD before but it doesn’t often do anything that STRANGE. I think.
the reply was to me
Oh, yeah. Clearly I’m bad at parsing indentation on mobile.
Just because it was not safe before, doesn’t mean it cannot be (alarmingly) less safe now.
And just because it logically can be less safe now doesn’t mean it is.
It is not. Not anymore. But I don’t want to get into political debate here.
I suspect parent meant it has never been safe
This isn’t true, as the US has been the steward of the Internet and its administration has turned hostile towards US’s allies.
In truth, Europe already had a wake-up call with Snowden’s revelations, the US government spying on non-US citizens with impunity, by coercing private US companies to do it. And I remember the Obama administration claiming that “non-US citizens have no rights”.
But that was about privacy, whereas this time we’re talking about a far right administration that seems to be on a war path with US’s allies. The world today is not the same as it was 10 years ago.
hm, you have a good point. I was wondering why now it would be different but “privacy” has always been too vague a concept for most people to grasp/care about. But an unpredictable foreign government which is actively cutting ties with everyone and reneging on many of its promises with (former?) allies might be a bigger warning sign to companies and governments world wide.
I mean, nobody in their right mind would host stuff pertaining to EU citizens in, say, Russia or China.
Which is to say: its not safe at all and never has been a good idea.
Nil channels are useful because essentially any practical use of channels should have a select with multiple branches and nil lets you turn off one branch of the select while keeping the others working. Even the most basic thing you can do, send work to some worker goroutines and receive answered data back, will have two branches for send data and receive data and at a certain point you will have sent everything and want to turn the sending branch off while still receiving.
You don’t really need
nilfor that, even if it’s the easiest ATM. The same thing can be done with an empty channel for select arms that receive, and a channel with a full buffer for send arms. The zero value for a channel could’ve instead been a valid channel with no buffer, which would satisfy both of the above.I’m personally not a fan of having
nilbe a valid value for some blessed types, it’s pretty error prone IMO (asnilis in general).Some of these things are historical patterns that have trickled down. Go really should be approached in the context of C and non-object oriented approaches to compound types. They were not trying to make an entirely new language with no baggage: they were comfortable with C and they just wanted something a bit better. If that’s a good thing or not is a different matter. I’d say for some things the defaults are not clear; some these things could be compile or runtime errors.
But much like in C a variable declaration does not initialize it (disregard zero values here); it’s also not a compile error to use an unitialized variable. Just like in C to both create and initialize things you resort to function patterns like
makeX–make(X)in Go – ornewX–NewXin Go. Just like in C passing an unitialized pointer to a struct as an argument to a function is allowed and the function decides what to do with it (e.g. passing anilslice toappend).In the Plan9 source code you find these exact same patterns in C which makes the decisions in Go much clearer. For example the function that appends to a string automatically initializes it if the string pointer given to it is
NULL. In the absense of a constructor-like approach this kind of late-initialization helps with the ergonomics when writing C. That’s why you also see the patternif x == nil { x.init() }in some Go methods wherexis a pointer receiver.Having that frame of reference also help make some “weird” things clearer, like why you can append to a nil slice but not assign to a nil map –
append()is a function and initializes thenilslice on the spot;somemap["key"] = "value"is a dereferencing operation and dereferencing a NULL pointer (thenilmap) makes no sense. A whole bunch of other things also start to make more sense in that headspace.I know why Go is how it is, and understand their decisions. Doesn’t mean I can’t disagree with some of them! “It is what it is” is not a good reason for status quo when designing, what at the time was, a new thing; especially when trying to make it fool/Googler proof ;)
Here specifically, the
chanstruct could’ve been designed so the zero value is an unbuffered empty channel, which makes sense for a zero value and avoids needing to addnilto the mix.Yeah I wasn’t necessarily writing to you in particular but more in general for people that might pass this by and maybe get a better way to look at these issues. There’s plenty to disagree!
Digressing into the second topic, I think they were well justified in not straying from the status quo when creating Go. I believe many of the decisions they made were actually very good and showed foresight, even if the language does not cover all the bases. But that’s a longer conversation.
I never thought about this. Thanks!
make(chan T, 0)indeed seems to work asnilwith my (simple) tests.https://go.dev/play/p/bnVOQnQMW-c
Your company needs to get rid of the silly junior / senior / staff distinction. It’s worse than the short-lived “rockstar” time, and I’m seriously disappointed that this disease is spreading world-wide. (Although given the zeitgeist, I’m half expecting full OR-1 to OF-10 soon)
The industry aligning on consistent, standardized levels across companies is one of the best things that we could achieve to improve software engineering as a professional discipline. I’m not asserting that the current zeitgeist is anywhere close to that, but it does at least seem to be moving in that way, especially in light of sites like https://levels.fyi and https://progression.fyi.
What is the actual problem that you have? What alternative do you propose?
What does that solve?
In what way do you see this “improve software engineering as a professional discipline”?
If the consistency and standardization comes down to having a very low granularity of adjectives attached to job descriptions, what’s the gain? I can’t think of a single way where the sole knowledge that Bob Bobson is a “Senior SE” and joins my team would help right now.
You didn’t answer my questions and responded with non-sequiturs. Your response assumes that consistent leveling is not possible, which is fallacious. If you start from that position, it’s no surprise you feel that way.
Imagine that meaningful, standardized levels could be achieved, and please try to answer my questions. Consider it a fantasy if you wish. But being unwilling to consider the possibility means you’re just entrenched into your preconceived notion, in which case there’s no use in continuing the conversation.
Your first question was what my actual problem is. And the main one is that I don’t really see any problem this vague level naming does beyond giving HR departments some rough guidelines. Does that improve software engineering?
In my opinion, meaningful and standardized can’t be achieved here, in a way that helps one for actual products. Especially if it all boils down to a single rank/title. This reminds me a lot of ‘70s Dungeons & Dragons, where you’re called a “Lord” once your fighter reaches 9th level. And of course said warrior is equal proficient at wielding longswords as they’re awesome with glaive-guisarmes.
But in the real world, skills are not equally distributed. And to continue my fractal of stubbornness, I’m also not a fan of ranking those on a five star or 1 to 11 scale, as that’s literally too one-dimensional. I’ve worked with a lot of Java developers in the course of my career, and still would find it very hard to objectively rate someone’s grasp of “the language”. What could be meaningful for a project with a narrow scope, focused on umpteen existing APIs and a need for a quick on-ramp might not necessarily overlap with something where you start from a blank slate and a lot of design work and prototyping is needed. And yet another employer might need someone focused on a lot of nitty-gritty details because code reviews and security checks are a major part. Are these separate, sequential levels?
That’s what it boils down for me: Our industry isn’t uniform enough to provide a solid foundation on which you’d build a standard for measuring developer ability. And definitely not something on the level of junior/senior, white belt/black belt or tenderfoot/eagle.
First, thank you for the response. This is well considered and does answer my question.
So, what I interpret here is that your belief is that the “junior/senior/staff distinction is silly” because we don’t (yet) have a way to make the distinction useful to us. You may even believe that it’s entirely impossible to do so.
I think for me the difference is that I am considering the benefits that would be reaped if we could achieve consistent, standardized levels that have actual meaning. I don’t think that it’s simple, and to be honest, I highly doubt if anything close will be achieved within the next 10-20 years.
I can say that part of the difference in the way that you and I think is that I am less concerned with individual skills and more with impact, aptitude, etc. So, carrying along with your Java example for a moment, I don’t really care so much about an individual’s grasp of “Java the language” in the abstract, I care about their ability to gain the level of grasp that they need in any language (given sufficient time) to achieve their task at hand. I agree with you that a different depth of understanding is required for different tasks, but I also feel that depth of understanding for individual pieces of technology would be one of the easiest things to standardize on (with better certifications, no different from how we decide that civil engineers know enough to be granted a stamp with the power to impact hundreds of thousands of lives), but I also think that level of standardization isn’t particularly useful.
So, this is a lot of words to say, I mostly agree with a lot of your thoughts, but I would add a lot of “yets” and “untils”, because I don’t think it’s inevitable that things remain this way. Thinking about software engineering as more similar to actual engineering is the shift that I am imagining could be so impactful to our industry, but I also don’t think it’s something that is going to happen any time soon.
The industry has no reliable way of grading people on those levels. One company’s junior is another company’s staff. The levels mean very little with no common base to agree on.
Plus it assumes a perfect level of meritocracy which doesn’t exist all around. I’ve personally seen very unskilled programmers — to the level where we had to fire them as the code was so subpar — rise to tech lead and then CTO in two years.
Abusing and subverting the levels doesn’t mean the levels are meaningless, it just means the meaning is being ignored. Saying someone was undeservedly promoted to tech lead and CTO inherently implies that the terms “tech lead” and “CTO” do have a proper meaning to you.
Even in a single company it depends on who the manager is and each person’s negotiation skill and drive.
I once participated in a technical interview for a senior dev (potential internal transfer) that couldn’t tell you the difference in complexity of searching for an element in an array vs a map. The worst part is I was asking those questions while not even senior myself at the time! It was a wakeup call that I really needed a raise and that the imposters are alive and well. This one came with a project director’s recommendation, the only reason we did the tech interview because on paper they were not a good fit either.
No reliable way yet, and your second paragraph is an indictment of exactly what happens when such standards do not exist, so you actually seem to be unintentionally arguing my point.
Something being difficult to achieve doesn’t mean we should throw up our hands and ignore it. Solving difficult problems is how we move forward.
Flat organizations have their own problems caused by the absence of title distinctions. It’s not the utopian destination you may think it is.
This. I’ve worked both in companies with extremely rigid and well defined levels and those that were entirely flat. At least in my anecdotal experience the ones with well defined levels were far more pleasant to work in. The ones with a flat structure develop hard to articulate or combat “implicit levels” that are subject to cliques and relationships. Humans pretty naturally develop hierarchies, so I’ve found it’s better when they’re designed and not grown.
[1] THE TYRANNY of STRUCTURELESSNESS by Jo Freeman aka Joreen, 1970-1972
Doing this for specific content in most websites is a cat-and-mouse game that only leads to frustration, in my experience. After a few months the HTML layout or the CSS class names of these websites changes and the improvements are undone. This is specially true for the worst cases in Web usability: websites with tons of JavaScript where classes are generated and re-named every deploy.
After feeling the pain I gave up for many years; what I’ve found works today is to just use “reader mode”.
I do the same: reader mode works great and is only a click away which is peanuts since I’m already planning to spend time reading the page. It’s also available on websites I don’t intend to spend time on again and it often bypasses soft paywalls.
I use Stylus a bunch on websites that are decent but unusable on a large screen (4k 32”) because their content is all the way to the left which forces me to turn my head, uncomfortable if reading for a while.
Those websites are generally not changing their classes often. Think Sphinx based docs, LKML, man pages, etc.
The article is a bit all over the place with its arguments but I agree with the general principle.
What the article doesn’t seem to focus on is to me the main advantage: composability. Every command in the terminal can be roped together with a common interface. Some people belittle that interface as “stringly typed” but in comparison to an IDE or other graphical programs, where nothing is composable, it’s very good indeed.
Not really related to the topic, but one of the things that I most dislike in Go is the heavy usage of magic comments.
I understand the idea that they want new code to be parsable in older versions of the language, but considering how heavily the language relies on magic comments I can’t see why it didn’t had a preprocessor marker like C and
#instead.But anyway, I know that the ship already sailed.
I wouldn’t stay it’s “heavy”, more like a sprinkling. There are only three magic comments that are generally used (build, embed, generate) and even then it’s sparingly. Grafana’s Tempo has ~170k lines of Go and only 5 of those lines are
//go:directives. A very large class of programs will never use any of them.I think that’s a pretty good deal.
It really depends, for example I have a small project of around 2k lines of code in Go and I had to use multiple build flags since the code is for a desktop application and I had to create different code paths for Linux/macOS/Windows.
It is even worse if you need to use CGo, since if you need to embed C code you basically need to write C code as comments. And since they’re comments, you don’t have syntax highlight at all (or at least I don’t know one text editor that supports it, and my Neovim does support embedded code since it works fine in Nix).
Now keep in mind that in general I don’t think it is a huge issue, but I still think the language would have been more elegant if it had something else to do those transformations. But again, this is Go, so maybe being ugly is better.
Quick reply, but it’s more about being simple rather than ugly. A preprocessor is even more magical and unwieldy than magic comments. C code often has endless runs of
#ifdef ... #ifndef ...to separate build platforms making the code mostly unreadable. Not to mention the magic macros that change the code from underneath your feet so you don’t even know what you’re reading.Even Rust’s macros allows one to create their own little language which no one else understands. Given how the express purpose of Go was to be easy to read for anyone not familiar with the codebase that’d have been a non-starter.
I figure the Go authors wanted, rightfully, to steer clear of the mess that is preprocessors.
Maybe I was not clear, I was not suggesting for Go to have a preprocessor, just that those kind of magic be indicated with a symbol different from comments. I suggested that it could be the marker of preprocessor in C (e.g.
#), but not that this is a preprocessor per see.The idea is because this creates a clean separation between what a comment is and what is something special interpreted by the compiler. Something that was different so it is easier to see that it is doing something, e.g. have different syntax highlight for it.
And the Linux kernel maintainers are just cool with this, huh? A disagreement over programming languages where the old guard refuses to give a single millimeter while folks are turning out high-quality, production code, leads to middle-school-caliber cliques, personal abuses, and predictably, maintainer burnout and resignations.
The Linux community is so much less with Marcan’s resignation, but it’s also so much less if nobody will rein in the LKML folks and maintainers, because this abuse will continue until Linux is an OS only for enterprise servers or things that can broadly look like them insofar as hardware support goes (see, Marcan’s notes about the death of the hobbyist hacker in the kernel and core low-level userspace). Linus - where you at, man?
I predict that if Linus steps in to issue a BDFL judgement, it will favor the LKML old guard.
Linux is already owned, for all practical purposes, by the LF member organizations, who are all big companies. Their common interest doesn’t favor technical innovation nearly so much as it does stability for existing business-critical application. So, yeah, enterprise servers, Android, and some of the embedded market.
That’s completely wrong.
Red Hat is funding the development of Nova, and graphics drivers are pretty much the first real world test for Rust in Linux. Microsoft employs several maintainers of the R4L core. Google pretty much was the first to look at Rust for a Binder replacement. All of them are also working on other projects adjacent to OSes in Rust (for example OpenVMM, Fuchsia, Coconuts Sam which was started by SUSE). So there’s a lot of desire for innovating with Rust among those companies.
I would argue that the Asahi GPU drivers were the first test, but that’s basically quibbling, they’re the first two for sure.
Yes, Asahi GPU drivers are part of the “graphics drivers” category. :) Nova is the second within the category of graphics drivers.
Graphics drivers are also obviously dependent on the DMA functionality, meaning that Red Hat has an incentive to find a solution to the current issue and it will help Asahi as well.
BTW the phone made a mess of CoconutSVSM.
My point wasn’t that LF members don’t want to use Rust in general (which is clearly false!) but that they may not have much shared incentive to advocate for Rust in the kernel. The R4L statements of support come from a much smaller set of orgs than the set of LF members. Funding or labor contributions aren’t as easy to find, but I’d be willing to bet we’re talking at least 2 orders of magnitude difference, and probably 3.
Though note that some of those companies supporting R4L are also some of the biggest contributors to Linux. The Linux Foundation itself doesn’t do much Linux development, and most LF members don’t do tons of Linux development. So a raw count doesn’t tell you much.
That would have been more true a couple of years ago, though. At the moment, those are the same groups coming under pressure to visibly start improving memory safety, and “we’re adopting Rust” is an easy signal to use.
At some point it’s easier to get memory safety features in standard C than to rewrite the Kernel significantly in Rust.
C23 didn’t standardize “attribute cleanup” but it’s not far fetched. It has memset_explicit which is a security-only feature.
As much as it is an important topic memory safety is clearly not a high priority for big players otherwise C wouldn’t be the most popular language for the past 50 years in OS development.
Rust makes a big deal out of it (and it is) but maintainers and companies are willing to trade all of it off for plain stability and compromise.
Not really. The same problem exists here as with Rust - working with upstream. Grsecurity had open source patches for decades with mitigations that reduced vuln impact radically, across entire classes of attacks. LF could have paid Brad to upstream but they didn’t for political reasons, and when there have been more politically minded folks like Kees who have tried it’s taken decades just to cherry pick basics, and that’s been met with massive criticism and public insults from upstream as well.
The problem isn’t Rust, the problem isn’t C, the problem is and always will be upstream.
Massive survivorship bias etc. All major kernels have been in C, thus those interested in kernel dev tend to learn C, thus new kernels are written in C.
So far, no one has managed to invent a good memory safety scheme that doesn’t require a lot of invasive changes to C or C++ code to work. Sean Baxter’s Safe C++ just added Rust’s borrow checking feature to C++, which ofc existing code structures don’t fit. I’m not aware of any proposal that would actually solve memory safety in C, just vague hand waving that it’s totally possible without too much effort.
I’ve thought a lot about this. I hate how complicated Rust is and I would love there to be a simpler way to get what Rust offers. But all the proposals to make C more memory-safe are laughably weak in comparison, and when I think about what it would take to draw the rest of the owl, I end up with something pretty close to Rust.
In particular, we are not yet (and perhaps won’t ever be) able to prove the safety of real-world C (or any other sufficiently versatile memory-unsafe language) in general. So if we are to have any hope of solving this problem we need a way to encapsulate things whose safety can’t be automatically proved. I think it’s fair to say this implies generic programming capabilities on par with Rust’s.
Also, while bounds checking is somewhat feasible to retrofit to C if you’re willing to make it slower and break all ABI compatibility everywhere, memory management is not. There’s a reason nobody uses attribute cleanup: it’s just not very useful. Heck. C++ has just about every memory management aid short of a borrow checker, and none of that seems to be enough. Rust makes this problem tractable by restricting what you can do, which feels bad to me coming from C, but we can’t statically solve the general problem today, and the runtime option is basically garbage collection. Practically speaking you cannot apply Rust’s solution to C either because you’d have to rewrite enough C to totally defeat the point.
Maybe we’ll get garbage-collected, bounds-checked C (I guess that’s what CHERIoT is) before we get a critical mass of software written in Rust. I honestly have no idea how to call it. But I don’t think there is any change you could reasonably make to C to create a path of less resistance than “rewrite it in rust” or “magical memory-safe hardware”, and the not-memory-safe hardware that’s in use today will probably go on being used for some time.
I think you could say the same thing about safety (the other kind) improvements in many fields. People who make and sell products have, historically, needed to be threatened with quite big sticks to spend money on making the products not hurt people. At some point I think some stick-waving will take place in software too.
I’ve had the “privilege” of building and maintaining a now 6-year-old C++ application. Originally built as a prototype in early 2020, we chose C++17 because it was the newest standard that was supported by the embedded Linux distribution provided by our vendor. The application is very data-heavy and allocation heavy (soft real-time processing of 25MP images at 20fps). It has gone through a ton of evolution and has had modules written by… politely… people who aren’t C++ memory management experts.
While the number isn’t 0, we have had very very few bugs that were due to memory safety issues. The keyword
newis not used explicitly anywhere in the codebase;std::shared_ptrandstd::unique_ptrare used heavily. Arrays and vectors are very rarely accessed directly by index but rather usingfor (auto elem: vec) {...}or other iterator techniques. We don’t, currently, have any static analysis beyond compilation in our CI process; at one point we did use SonarQube and it did a good job of pointing out some questionable stuff in some of the early code.You’re right, compared to Rust, our code has not been perfect from a memory-safety perspective. At the moment, through its current execution paths, it appears to be, at least according to valgrind. We’ve soak tested it for weeks on spare hardware and don’t seem to have any memory leaks. We’re doing thousands of allocations/sec with small objects, and enough large allocations every second to consume all 64GB of system RAM in about 2 minutes.
Might there be issues lurking somewhere? Maybe. From an engineering perspective though? This system works fantastic and if there are issues related to memory corruption they are incredibly rare in practice.
It’s worth observing (as more follow-up/color than any kind of “rebuttal”, since I don’t really disagree!) that broader standards like C23 are less relevant for projects like the Linux kernel. There was some early activity in the 2009 to 2013 era more based on clang analyzer & warnings, but it probably wasn’t until late 2013 about 11.5 years ago that they even regularly tested with
clang. From 1990 until then (aka a quarter of a century), the Linux kernel was basicallygcc-specific C. Any compiler that wanted to work with the Linux kernel was/is beholden to plenty of gcc-extensions like its inline asm syntax. So, the reality today is probably “compatibility with whatever extensions both clang & gcc support”, not any specific standard. (EDIT: and I make no claim as to “how close” to “good enough” various notions of “safety” one could realize this way. It’s more a point about C23 relevance vs. compilers willing to experiment more than a standards committee might be. A related observation might be that even kernel-specific C extensions for new static analysis likesparsehave been considered fair game.)I don’t doubt it, but could you elaborate about the pressure? Who’s exerting it, what leverage do they have? Maybe provide some examples?
As an example of pressure from one direction, the NSA report on memory safety from 2022 led eventually to a National Cybersecurity Strategy with a pillar about “shifting software liability” and USG investment in “memory-safe languages”. And this was repeated by all the Five Eye governments. That sort of thing implies it’ll be on the agendas of regulators, procurement officers, investors, and insurance companies, which puts it on the agenda for CIOs and CTOs.
We also have people like Mark Russinovich (Microsoft Azure CTO) saying C/C++ should not be used in new projects, and Microsoft does quite a bit with Linux nowadays.
The EU introduced product liability rules recently that may be a big redirection of priorities for embedded and appliance software folks.
Several governments, chiefly the US federal government. Also, while it doesn’t directly mention memory safety (I believe), the EU Cyber Resilience Act does mandate stricter cybersecurity standards and puts heavier liabilities on companies for faults in products with software elements. The oft-quoted figure that 70% of vulnerabilities are caused by memory safety issues is presumably not universal, but if it is true for Google and Microsoft products, then it seems like they have a lot of incentive now to make sure their products are memory safe.
Alex Gaynor’s What science can tell us about C and C++’s security gave the following numbers back in May 2020:
That’s basically every OS and browser engine with significant market share. (If anyone has cited numbers on the BSDs, please let me know. I don’t have time to go digging myself.)
many of those projects also use tools like valgrind regularly in their builds from what I recall. It really looks like at some point or size in a codebase if it is C/C++ you are almost guaranteed to have memory safety vulnerabilities no matter what you do to prevent it.
I prefer to describe it as “Regardless of how capable of writing correct C or C++ individuals are, it’s been shown that people in groups cannot”.
It’s more diplomatic, given how emotionally invested in their own skills people can get.
…and more recently, I’ve started describing Rust as a lockout-tagout system for software development, based on the ways its designers chose to focus on maintainability of projects written in it. (eg. working to avoid the need to reason globally, providing a powerful type system for encoding invariants, etc.)
Honestly, Linux won’t be the dominant “new” server in a few decades and it’s exaaaactly because of the way upstream is supported. Linux will be relegated to “weird” hardware that IBM pays a fortune to maintain and provide enterprise support for.
He’s a huge part of the problem. Always has been.
Where is this? I cannot find
I think the most important feature for me will be tracking tool dependencies with the
gocommand directly. No more problems caused due to different tooling versions when collaborating with other people :)I was lucky that most tools I am using for my projects can be used as libraries and can be used with my own little command line executable, which then can be used in a versioned manner. Here is an example where I’m using
//go generatewhich wraps an internal command, which usesesbuildfor building JavaScript and CSS assets.One possible downside is if the tool dependencies clash with the dependencies of your package.
I’ve never had this problem, but you can use a separate mod file for tools if you’re worried about that.
Although I wrote a post, being very excited about
go tool, this is still an unfortunate side effect of the pattern. Using a separatego.modfor them works, but has it’s own tradeoffs 🙃That’s a possibility but you can always go back to what people were doing before with a
toolspackage or doinggo install @versionon its own.I haven’t run into an issue with the common dependency graph yet. The unfortunate thing about it is not being able to discern quickly your own dependencies from the tools’ in
go.mod.In fact I seem to have problems now with different tools having conflicting dependencies
I’m confused as to why an HTML blog would strain a server more than a plain text blog. Serving this text as HTML might, POSSIBLY, double the size from 1kb to 2kb. Killing off the favicon would be just as effective, maybe moreso, for saving server resources.
I had assumed this was satire.
Anyway it’s not even close to double I think. I wrote a 6 paragraph blog post, one with and one without html. The html was just
Then
<p>tags aroudn the actual text.The end result was just a 400 bytes smaller in plain text. But if you put links in you could click them etc etc.
Also, txt in a browser is surely more likely to be deprecated over html
Honestly, you’ve gone way overboard with the HTML there, too. You could drop the lang, the doctype, and the entire head block and it would work fine on 99.99% of browsers.
I think in HTML5 he could drop everything except the doctype and it would be valid, wouldn’t it?
The smallest “hello, world” HTML5 document I am able to write that passes both the Nu HTML Checker and
tidyversion 5.8.0 is the following one:Anything less and the Nu HTML Checker emits a warning. Tidy output looks good for the above HTML, if we don’t mind the following warning about implicit tag:
Dropping the lang is bad for accessibility. Without the doctype it’s not valid standards mode HTML5. The charset is necessary if you have any non-ASCII characters. The viewport tag is needed for proper rendering on mobile.
Dropping the lang is no worse for accessibility than just serving a txt file.
(I will go to my grave arguing that “lang” does nothing for accessibility, but I’ve lost that fight here already)
I don’t think txt files are so much of a problem in term of accessibility (I mean you can configure your screen reader to change the spoken language).
I created this txt blog for two reasons :
But I really like lightweight websites with html & css, and love to waste my time exploring funky lightweight designs from 1MB club (https://1mb.club/) :)
To be clear, you should do what makes you happy, and if this is it, good on you. I just don’t think the savings are significant, for the tradeoffs.
While I understand the spirit, you may be shooting yourself in the foot. If your intention is for people to read your articles, not just have the server serve them, you’d do well in improving the reader’s user experience just a bit. Plain text is just not great to read on desktop and on mobile it is unreadable (I mean literally unreadable because the font size is miniscule).
As it stands it’s a bit contradictory to care about readers not being able to see your website if it gets squeezed to death because when they are able they can’t read it anyway.
I don’t really mind if no one is reading my articles. I have another blog (in french) at l3m.in, which is made of html, css and even some images.
I disagree on your take on mobile readability; the reading mode (at least on firefox, haven’t tried on another browser) handle txt files just fine :)
Not as practical, but I also made https://1kb.club when I thought 1MB wasn’t extreme enough ;)
Oh you’re the one behind 1kb.club! Thank you for your work, I love stumbling across your site from time to time and discover new websites!
Holy crap, I had never heard about this “duel”. The way the author sort of debases Jeffries’ entire approach is something else:
That’s harsh, but quality writing. Reminds me of Hunter S. Thompson’s Nixon eulogy. May I never be the target of this sort of breakdown.
A related but much kinder breakdown is McIlroy vs. Knuth, which is perhaps the opposite of this one (a critique of too much design).
If you enjoyed the McIlroy duel, you (or someone else reading these comments) might like this lovely context article, and this light response to that one.
That’s pretty good. It’s good to embrace those and I think it’s important to highlight you can’t always be all of those at the same time.
I’ve felt guilty when my code doesn’t check all the boxes these hats help with, but more and more I realize it’s a foolish errand.
Seems like a cool use, but I wonder: wouldn’t copying and pasting the schema SQL to an LLM prompt and asking it to write a query yield the same results? Or uploading the schema file as some LLMa allow. I don’t get why being sort of built in the DB makes it better.
I actually never thought of doing it but seeing this it seems like a great way to write pure SQL in more complex database schemas.
Yes, you’d get the same result.
That’s what this tool is doing for you, but with a more convenient UI.
Exactly! Litecli is making that request to the LLM with the database schema and parses the sql query from the output to make it easy to iterate.
The intent is to keep you in the flow and not have to switch to a different tool to do what you want. Basically reducing the friction in the workflow.
I would also clarify that this is not built into the DB. LiteCLI is a separate client for SQLite that added this workflow.
I agree with the points on code and project organization but I’d say these are more Rails/Ruby-oriented points and not a general issue. The main problem is that Ruby itself gives us very few tools to organize code: there’s no package-level segregation where one can think of packages like a separate unit from the rest, calling code from different modules is syntatically hard, plus the Rails community focus on arranging projects by taxonomy rather than subject (“models, views, tests” and not “authn, search, accounts”) means every folder is a grab-bag of different things that don’t belong together.
Different solutions have been tried over the years but they all depends on people following unwritten conventions. In all large Rails projects I’ve been on cross-cutting concerns show up everywhere in the codebase like a virus and it takes a lot of discipline to prevent it.
This isn’t a general problem that feels the same in every language though, it’s much less of a concern in Go and Rust for instance.
How curious, I almost went to work for Hubstaff. I eventually declined the offer for a different place but it seemed like an interesting company. However I can’t even begin to imagine how mundane and unimportant software work must feel in the midst of a war…
Well, yes and no :) Of course, it was a talk for a Ruby conference drawing from my mostly-Ruby experience (I probably should’ve included Ruby specificity in the title to post here). But I believe that the set of ideas/directions of thinking I am talking about (looking at code as text/stories, focusing on “truth” and “telling how it is”, attention to what “page of code” can tell, etc.)—even if sounding somewhat naive/idealistic, can be applied more universally. I’d probably choose other ways to illustrate it and build a narrative if targeting a more general audience. I’d actually probably be in the future :)
Yep, pretty surreal at times! Not as much now (I am in relative safety, performing tasks not completely unlike my “civil” job, though much less mundane), but the first months in the army were quite wild. Reviewing code that handles things like “refactor the representation of a specific task tracking metric for small US companies” on a satellite connection in a heavily bombed frontline settlement between other duties… Fun times.
But the interesting thing is that you remain your old self even in those conditions. I started working again just a few weeks after the start of the full-scale invasion (when still a civilian/volunteer but in a city that was then very close to a frontline), and in a few months, I was able to be again interested in software development and start writing blog and trying stuff when I had time. Humans are weird like that.
You’ll maybe laugh, but I did present Padrino as an alternative to Rails, for those that want different taxonomies at a conference and literally got shouted at by someone in the Q&A for what my problem with Rails is. I mean… the whole point of the talk was that there’s different tastes and Padrino caters to a different one.
Another issue I ran into was that I worked once at a company that did use Padrino and I did build a project setup that was similar to how you want it, using Padrinos mountable apps feature (Imagine a folder per concern, with an optional Rack stack in front of it). I got into a long discussion with their tech lead in which he was telling me I was not using the feature as intended. I’m the author of the mountable apps feature…
It’s sooooo hard to get these things out of peoples brains. Rails scaffolding was an amazing feature to show that webapps could be built swift. But it came with such an amount of damage.
I love Ruby, Rails was always… something I had to deal with.
One of the most important things I learned from school was from TA’ing a class, when I realized that different peoples’ brains can trace different paths through the same terrain. Go far enough and there’s seldom a single optimum path, even with the same starting and ending points.
I hadn’t seen Padrino before. Cool. Thanks for sharing.
Happy to. TBH, nowadays, check before you buy. Padrino is maintained and has a few gigantic deployments no one talks about (and I can’t), so it will stay around. But there’s also Hanami and all the others out there.
But I think it still strikes a super good balance between helping you, but also being able to pop open the hood and figure things out yourself. The whole project is is built against a stable and documented internal API that lived for a decade. So for example a newly generated project just generates you a
boot.rbthat you can actually follow and rearrange yourself. And because the API is stable, your re-arrangements will survive version upgrades (or… it’s a bug!).It’s the project where I learned a ton about collaboration and I will always be thankful for that.
Laravel does this too and it pisses me off
Several large Ruby shops have created package tools like you describe. Shopify released theirs as packwerk. I haven’t used this one, but it looks very much like the one I have, perhaps due to employees cross-pollinating.
These are more generally called build-time (as opposed to run-time) dependencies. While tools are the most common use-case, another example of a build-time dependency would be a plugin for the build system. The main reason why we need to distinguish between the two types of dependencies is because we may need to build them with different configurations (for example, optimized for build-time and debug for run-time) or even for different targets in case of cross-compilation.
I can see why all the tools would share a dependency but also forcing the run-time case to use the same version is strange. Simplicity of implementation seems like would be the only reason. Strictly speaking, even tools (but not build system plugins) need to share the same dependency since they each run in a separate process.
No, these are not build-time dependencies. When you run
go build, these tools do not run. They are development-time dependencies. The programmer simply has these tools at their disposal to run for when they are developing code. It’s not a build-time thing.Thanks for the clarification, I missed the part that the tools have to be run manually.
I feel conceptually development-time (I would call it development-only) is orthogonal to build/run-time: you can theoretically have a development-only run-time dependency (for example, some instrumentation support library which is only useful during development). So this Go tool thing would be more accurately called manually-run build-time dependency (which is necessarily development-only). It would still be built for host rather than target when cross-compiling, right?
Yes, they would be built for the host.
I think this misses that the tool run time (and number of runs) vary drastically from build time and number of builds. Eg generating code for protocol, which might have happened once and then been rebuilt on hundreds of different commits. I’m not sure l calling this build time is all that useful.
What Go has is a special case of the following more general arrangement: a build-time dependency with its output stored in the repository (BTW, a practice which is frowned upon by many for valid reasons). In this setup the tool will only be executed (by the build system) when the input changes. Otherwise, the build will use the pre-generated output, even for a freshly cloned repository.
In the Go case, the “update output when input changes” part is handled manually, “for secure” I assume (or maybe “for simple”). But I still think it’s helpful to use more general terminology.
While I appreciate it I think your focus on terminology is making this more complicated than it has to be.
go get -toolis essentially a package manager, nothing more than that. These tools might not be dependencies at all or might never be ran on any machine, development or otherwise.Rather tangentially:
A little further and we have macros! Though a lisper, I miss macros in Go when I’d like to precompute things, load data into the binary etc. As is, it sadly demands a separate prebuild step, computing and placing such information in .go files, which can then be compiled.
Does the
embedpackage not work for that? Doesn’t involve a separate prebuild step and exposes the data quite nicely through the standardfspackage. I’ve used that to embed, eg., templates and other external data.Embed only takes in e.g. file contents, but doesn’t let you preprocess (e.g. label, sort etc.) during compilation.
I agree with a lot of this. I actually think the article is being too kind, in the wild I’ve seen that many Rails apps degrade into more and more CPU-bound, mainly due to coding decisions that they’re more or less expected to make (the Rails Way).
I’ve worked on some projects were rendering out the template was about an order of magnitude slower than the DB query itself.
I’d also argue a lot of the database issues are because Rails divorces you from the DB layer too much and favors more inefficient database design decisions (no materialized views, etc).
Sidekid ends up being the dumping ground for workloads that are too slow to do in-band on the request path. These include converting files, performing calculations, but also things that look IO-bound but have CPU-intensive work due to Ruby being slow such as loading thousands ActiveRecord objects in memory and iterating over them.
Many of these companies actually take performance very seriously and were probably commiting fewer mistakes. I imagine the speedup would be even more significant in some companies without that know-how. That is to say the average large Rails app is in worse shape than Shopify’s.
I’m curious about how you measured that. Several tools show view rendering time as slow but it was because the database fetching only hit during the template rendering (because Active Record is lazy)
I guess that part got confusing, I meant for that sentence to be read in conjunction with the first part about Rails leading one towards inefficiency. Yes, it was because N+1 queries and lazy loading popping up in the render phase. My point was that while those are problems in the database interaction the DB itself was not an issue, just the misuse of it which was mostly caused by following Rails recommendations. The ideal query eventually ran ~10x faster after a different schema design.