The new Arc implementation, Clarc, is in Common Lisp, so it can use SBCL, which I wouldn’t call a JIT (there are occasional debates.) Clamp was a similar Arc->CL portlet in the past.
I agree, it also seems to be applied rather haphazardly, and generates bad feelings.
I think if content is “self promo”, it just won’t be upvoted by readers – it’s not clear it needs to be a special case rule
(maybe controversial – but I would not mind a downvote button for both stories and comments. You could downvote stories that you think are self promo, as well as accusations of self promo that are “tired” … i.e. it feels like the voting mechanism can solve this problem, without a special case, and without a lot of distracting discussion. The downside is that it can generate “why are people downvoting me?” discussions :-/ )
Where was that proven? Can the mods weigh in on any abuse of flags?
Personally I think the biggest flag missing is “low effort”, like just repeating a joke you can read on Reddit, low effort opinions on Go and Rust, dismissing the article based on the first paragraph, etc.
I’d also be in favor of a “self promo” flag, if that can reduce tedious back and forth about what is self promo
Determinate Nix - soft fork / downstream of cppnix. they insist it’s not a fork, but it functionally is
Snix - built from scratch, with a very different architecture, but the goal of eventually being compatible with cppnix
Tvix - now renamed to snix
Devbox - a wrapper around nix, currently using cppnix
Guix - another completely different project, using similar concepts, but the scheme interpreter
In practice:
You’re unlikely to use snix for now, unless you’re an existing nix person and are looking for trouble / fun. Maybe in a year or so.
You might use devbox if you want to use nix (the package manager), without using nix (the language)
Cppnix, Lix, and Determinate Nix can be swapped out at will, and will function the same on 99% of codebases, except for some extra config flags on the first two to enable ‘experimental’ flakes.
The first three are separate for social reasons mostly, the others exist because they want to take a different direction in the actual software.
Lix also has one extra experiment: pipe-operator since v2.91 (August 2024).
I love that feature dearly and use it in my personal config to avoid deeply nested expressions: h z (g y (f x)) == f x |> g y |> h z
Not saying a pipe operator is bad, but we have an option now with ‘good enough’ ergonomics. Does the pipe operator offer some performance gain by being in the language?
I don’t even want to translate it to lib.pipe, it would not be much of an improvement over the original, pipe/operator-less form. I do not find lib.pipe’s ergonomics “good enough”.
I used lib.pipe before enabling the experiment, and I like that function too! I just like the pipe syntax even more.
I’m using it in my personal config, which no-one else ever uses/sees, just for the syntax. And lib.pipe is the obvious, and easy to switch to, off ramp if I need it in the future.
Performance is one of the RFC’s unresolved questions and I haven’t noticed a difference because most of my config eval time is not spent in lib.pipe.
I don’t want to duplicate the whole RFC or its discussion here, but having the functionality in the language has other UX benefits like better tracebacks.
There is reason to expect that replacing lib.pipe with a builtin will reduce its overhead, and that the builtin should have little to no overhead compared to regular function application.
This is precisely the advantage I could see if it were a built-in—otherwise the value add is pretty slim to keeping it simple just sticking with lib.pipe (& I say that having used pipe-like operators in PureScript, Elm, OCaml, Haskell, & Nickel and a fan of operators). Recently tho I’ve been coming to the opinion that our function notation sucks & should be post-fix by default ala reverse Polish notation.
It did, it took the Nix daemon and modified it. For some time they were still subject to the same CVEs. There’s been an effort to rewrite the daemon in Scheme, but no idea how that’s going (maybe it’s done?).
For some reason a few Guix-affiliated people refuse to acknowledge that they forked the Nix daemon, it’s probably something political.
The nix daemon is a very small piece of the overall system. That piece was borrowed, indeed, but 99% of the system is not that. It is an otherwise completely separate implementation of the functional package management paradigm that makes some radically different design choices. The daemon thing is a distraction. Guix is not a fork of Nix in any way that matters.
I did hear about Linux distros. I think the distro situation has squandered thousands of hours of human effort over pointless minute differences and archaic 90s software design choices.
The politics are a little sour here and there, which is probably the biggest downside. Otherwise, good point. The Lix and Snix ports seems really healthy and forward looking. Maybe it causes some confusion for non-NixOS users? But in NixOS, most probably use whatever they are fed by the base system.
I think the danger is that they might start diverging on Nix-the language and we get incompatible Nix repositories. This is already the case to some extend with e.g. flakes (though cppnix/lix/determinate all support them), but could get worse if implementations start adding implementation-specific features.
I think this risk is real, and is currently mitigated primarily by one thing: The steadfastness of nixpkgs maintainers in refusing to accept code in the packages that doesn’t work in Nix 2.3 - which is ancient at this point.
What I can see happening is a sort of Cambrian explosion of Nix implementations and dialects, but with a shared minimal viable subset in “whatever is needed to use nixpkgs”. Sort of like the situation with C, where a bunch of compilers all implement the ISO/IEC standard but also offer a ton of nonstandard extensions, with nixpkgs acting as somewhat of an ad-hoc standards body.
You also don’t need to “shell out”, because YSH is already a shell! Both traditional Make and Ruby Rake delegate to the shell, which makes escaping non-obvious, among other things
Thanks for noticing! I would definitely like more Ruby users to check out YSH (feel free to join https://oilshell.zulipchat.com/ or send other interested people that way)
A slogan is “YSH is for Python and JavaScript users who avoid shell” … But that really includes all of Python/JS/Ruby/Perl/PHP/Lua (though Ruby and Perl users don’t seem to avoid shell as much, which I think is good!)
Partly this is because I know Python and JS pretty well, but not Ruby. But I’m interested in and respect Ruby, e.g. I generated some “survey code” as part of the YSH language design, e.g.
The C source for this Thompson v6 shell is quite archaic, using =+ instead of += and doing a very funky syntax tree representation. Understandable since C itself was evolving (being evolved by the same person, actually), but has anyone ported it to something more modern like ANSI C 89?
I agree, though that link you have (same guy/website as TFA, actually) only links to enhanced tsh / etsh or “osh” sources that I can see. The very oldest release from 2003, osh-030730.tar.gz, is starting to look closer to the structure of the v6 source version, but still has a lot more distance than I was thinking a “port” would have. Anyway, THANK YOU!
xv6 is a re-implementation of Dennis Ritchie’s and Ken Thompson’s Unix
Version 6 (v6). xv6 loosely follows the structure and style of v6,
but is implemented for a modern x86-based multiprocessor using ANSI C.
So they both do 2 recursive invocations – to runcmd() or execute(), on the left and right node of the AST.
Oh weird, but actually now I notice a difference … xv6 is less efficient?
It has a pipe() and 2 fork() calls for every PIPE node
original Thompson shell has only a pipe() call, and two execute() calls, which may or may not fork ???
So basically in xv6:
cat | cat | cat | cat
is executed like:
cat | (cat | (cat | cat)))
That is, you would expect there to be 4 cat processes to run, but there are at least 2 more processes.
But not Thompson shell? I’m curious if I’m reading that right …
BTW https://oils.pub/ stores all the pipeline components in a list, so it doesn’t have this left/right recursion. I’m pretty sure most shells, like bash and dash, are like that – you can have a variable number of children in an AST node, not just 2
The xv6 shell also has a different syntax - no backslash escaping or any quoting and no $vars and probably other differences. It is also very different in structure and function, as you observe. It is more a re-implementation of an “even more simple” shell-like thing than even a “loose following” of the v6 shell. I’m unsure how many similar statements might apply to other aspects of the xv6 project.
EDIT: btw, yes you are reading it correctly. I patched & compiled the xv6 sh.c to just run on Linux and then ran it as ./a.out and then just copy pasted cat|cat|cat|cat with the programs waiting on the terminal and ps (well, really my pd ) output looks like this:
If you read the OG v6 shell source in TFA, you can see these things handled (at least $0 $1 and $$). { Might well all be just academic/historical/hair splitting, but that’s sort of the whole topic, too. :-) }
Through the whole thing a huge [citation needed] was glaring so bright I barely could finish it.
Every successful business started with those languages eventually rewrites their codebase to use one of the “lower level” languages because big codebases written by many people are hard to maintain and modify without the support of a static type-checker.
[citation needed]
GitHub, the very site it was published on is written in Ruby. By the OP classification it’s a level 4 language. As of right now GitHub is 17 years old. GitHub is also 6,500 strong company. Granted, not everyone is working on GitHub but even if it’s only 20% it’s still 1,300 people working on a Ruby project. they seem to manage, more or less.
Shopify is another big Ruby app. Founded 2 years before GitHub (so 19 years old). At this time about 8,300 people work there. By the same 20% assumption, it’s 1,600 people working on a Ruby app. The seem to manage even better than GitHub.
These are not the only two examples out there. It’s the two I think most of you are familiar.
So when does eventually come? Or at what point the team is too big?
And so we come to the levels 2 and 3, where most professional programmers today spend their time.
[citation needed]
I would wager a solid $1 that JS, Python, and PHP alone employ more developers than every other language combined. I’m also fairly confident that code in those languages is produced at higher rate, as well.
The only gap between them is that interpreted languages can include “eval” and dynamic meta-programming (modification of program structure at runtime). These features are usually shied away from in production code though, and are more helpful during development, especially for testing.
[citation needed]
Rails (the thing GitHub and Shopify are written on) is arguably a skyscraper of meta-programming. Every Rails-like framework employs meta-programming to some degree. It wades a bit into the weeds of semantics but Spring (Java framework) annotations are a kind of met-programming, so are Rust #[derive(...)] and other annotations, even C pre-processor. Meta-programming is all over the place and was since the dawn of programming. It’s in every production system even if it doesn’t support eval at runtime.
The best part is that all three languages share pretty much the same syntax, and they are built so that calling from higher level to lower level variant is effortless.
[citation needed]
I’d argue that different syntaxes at different levels is more beneficial as it helps switching context. You’re in for a whole lot of… adventures when you think you’re writing one language when in fact you’re writing a completely different one.
Take this “RustScript” for example:
let rect1 = { width: 30, height: 50 };
There’s a whole lot of questions I have about this line. How do you map this onto a lower level Rust? You have to map it onto the lower level somehow because it’s gotta be “effortless”. But how do you infer a type of this? Is this type equivalent to every struct with those fields? How do we convert between the types if the conversion is not defined in the lower level?
Can you do it in RustScript? It’s supposed to be dynamically typed. What would happen to the type? Can you still use this awesome rect with low-lever Rust-proper?
This is just from a single line in the example. I’m sure there will be much much more questions when expanded to the whole of Rust.
Now, don’t get me wrong, I appreciate the idea. It can be an interesting thing to explore. I’m just a bit miffed that OP is backwards. It makes some dubious claims to set the stage for the idea and does no exploration of it at all. I’d rather it started with the core idea of language sets and explored it in more depth. Instead of stripping down Rust syntax to look like JS I’d love OP think about interop (arguably, the core feature of a language set) and how it might work.
I think Shopify is a Sorbet user; if so their codebase moved from level 4 to level 3. Dunno if Github uses Sorbet.
Re. metaprogramming, the dynamic qualifier is important. I like to write code that writes code, but that’s almost always static codegen. Code that rewrites itself is far too brainbending!
Yeah I think the opposite direction bears just as much fruit :-)
You could start with the most ergonomic/productive thing and make it fast – in contrast to starting with the fast, complex language, and simplifying it.
We did that with https://oils.pub/ – started with Python, and then statically typed it with MyPy, and then translated that to garbage-collected C++
So we went from tier 4 to tier 3 to tier 2 !
bash is written in a tier 1 language – C. And OSH is faster than bash in many cases, although there are still cases where we’re slower.
We are able to be faster because bash is a very suboptimal program! (e.g. it uses a ton of linked lists, not sure there are any hash tables at all) Most big programs are suboptimal, especially big C programs.
Wow! I really want to know how the images were made …
I do wonder about AI, but at least when I tried the AI generators more than a year ago, they couldn’t produce anything like this. They produced stuff with a very distinct style, more like this - https://www.youtube.com/@protocoltownhall/videos - everything there is clearly AI
Many of the images look photo-realistic, but then I feel like this page would have taken like 2 years to make, which seems implausible (?)
Oh well now I see she is working on AI, so I guess it must be AI … that was not my first thought, it seemed almost like the opposite aesthetic - https://wattenberger.com/
Or maybe it is some advanced Photoshop plugin that does style transfer of the “Yarn”, but many of those plugins work with AI techniques
(Also, the plea for interfaces that use our senses of course reminds me of https://dynamicland.org/)
i was very impressed with this page when i first saw it, but i refrained from sharing it here, as the author admits to using AI to generate these images, on bluesky: https://bsky.app/profile/patrickpietens.dev/post/3lk4qz7gg7k2n (on the first reply there)
it becomes more obvious towards the end (upon observing the closeups of each characters’ faces), that there is no similarity from one image to the next (the yarn pattern is of different sizes, the backgrounds are blurry and meaningless etc.)
With respect, I would disagree on the effort part. Yes, they used AI to generate a bunch of images. Throughout history we have used tools to make us more efficient. It seems to me that this is not all that different from that. They had an idea and wanted an article on it, but might not be that much of an artist OR didn’t want to spend the tens or hundreds of hours it would take to make this.
I’ve not managed to get these sorts of results myself (nor have I really tried, in fairness), but there was an AI art Turing test that did the rounds a few months back that showed some pretty stunning examples of AI-based art that doesn’t just look like standard art.
I have no idea what the actual process looks like for generating these images, though, and can claim absolutely no expertise in this matter.
EDIT: And just like you, the main thing that made me realise the images were AI generated was the amount of time it would have taken to create them if they were real, although after that point it became easier to notice things that were a little bit off, like odd features or weird blurring. Although maybe those would have also been present in real photos, and I was just primed to notice them because I knew the images were AI?
Interesting article … I think this page looks a little more surprising than those examples, probably because it’s an experienced designer + AI, not “one shot” image generation.
Although I don’t know exactly how all those were generated … but definitely a few of them toward the end have the giveaway “AI style”
And yeah I am judging it based on style, and I know AI can generate multiple styles … but yeah I think this is a good example of augmentation/acceleration, just like you’d see with coding. A random person is not going to get as good results from the LLMs as an experienced programmer!
In this context it’s useful to distinguish between an AST and a concrete syntax tree: a CST includes all the details of spelling, spacing, and commentary that are omitted from an AST.
A parse tree or parsing tree[1] (also known as a derivation tree or concrete syntax tree) is an ordered, rooted tree that represents the syntactic structure of a string according to some context-free grammar.
I checked the terminology of the Dragon Book specifically for this comment:
“parse tree” appears in the index, but “concrete syntax tree” does not appear in the index
“Syntax tree” refers to “abstract syntax tree”
So the Dragon Book is making what I call the CST vs AST distinction, with slightly different terms.
Additionally, I make a “lossless syntax tree” distinction.
The key difference is that the Dragon book only talks about compiling, not formatting or translating. I think “lossless syntax tree” is useful for talking about what Clang and Go do – they are neither CSTs nor ASTs.
Their trees don’t “recapitulate” the grammar – quite the contrary, their parsers are hand-written. But they contain all the physical details of the code – not just parens, but also whitespace and comments.
It is not a good formatter, though, because it cannot enforce a maximum column width.
You can fight me, but this is exactly what makes gofmt excellent, and rustfmt very annoying. zig fmt is best though, there, the user controls line breaks via stuff like trailing commas.
I see from the upvotes there’s a perspective here, and I’d like to understand it.
For contrast, here’s my perspective, from which rustfmt is good and gofmt is bad.
I like to view code on a vertical monitor, because seeing ~two screen-fulls of text at once is much better than just one. Or, if I’m using a horizontal monitor, I like to have windows side by side. Either way, this (together with having relatively big text) limits my screen width to a little more than 100 characters of width. This isn’t unique to me, lots of people like to limit line widths.
Soft-wrapped lines are awful. Hopefully this isn’t a contentious point. (By soft wrapping I mean when you have e.g. a line of length 150 and a screen width of 115, so 35 characters get shown on the next row.)
Therefore it’s very valuable to me to ensure that all lines are no longer than 100 characters (or 120, or 80, or whatever).
This leaves two options: (A) have a tool like rustfmt pick the line breaks, or (B) have developers pick the line breaks, like gofmt has them do. Option (B) allows artisinal hand-crafted code formatting, with three downsides: (i) it requires more effort by developers that to me feels like busywork, I’d rather focus on the code itself, (ii) it’s going to be less consistent than what an automatic code formatter gives you, and (iii) realistically not everyone is going to obey any particular line length limit, and there’s going to be soft wrapping.
rustfmt routinely produces suboptimal formatting, where things that
should be multilined are crammed into a single line, and things that
should be compact are unnecessarily spread out. A related problem is
that formatting is self-inconsistent: whether something is multiple or
single line depends on an arbitrary threshold, and not on the
formatting of three logically related expressions right before. Given
that the code is read way more often than written, it is a good trade
to spend a little more time writing it, to make it more readable, as
long as human only chooses formatting, with the tool doing the
actual job of adjusting whitespace. Putting just a little mind into
formatting is especially useful because often the best way to format a
long expression is to break it apart.
The tradeoff here of course depends on a particular team, as choosing
a tasteful way to present your code requires taste!
You still of course want to enforce line length in CI.
I suspect a lot of the difference in perspective is that my experience involves a mix of:
Lots of experience with rustfmt: learning to expect what it does, and coming to value its mechanistic predictability. For example, knowing that if function arguments are split across multiple lines then there’s exactly one per line makes a lot of things easier (e.g. comparing args across functions, re-ordering args).
Absolute anarchy, with e.g. a coworker who defended having 200-char long lines, and a founder who couldn’t agree on any set of variable capitalization conventions that could be enforced by clang-format because the boundaries he wanted to draw weren’t syntactic, resulting in no consistent convention at all.
Having co-workers who present code tastefully sounds nice too.
Good point about enforcing line length in CI, that completely mitigates my point (iii).
I am not sure, but I think the original comment might be complaining more about “over-canonicalization”, which we discussed last year.
rust fmt is basically a syntax tree pretty-printer, from what I understand. It clobbers your formatting.
go fmt is not. It fixes certain things about your formatting, rather than clobbering it.
So I think this part of the article is wrong:
Formatters are source-transformation tools that parse source code and re-print the resulting AST in some canonical form that normalizes whitespace and optional syntactic constructs
It is leaving out the go fmt / clang-format / yapf / scala fmt type of formatter, which we also discussed. They do not re-print the syntax tree; they use some kind of tree to fix the text.
And I’d claim that go fmt is the tool that made formatting “universal” among working programmers (although I don’t use Go myself, so I haven’t really used it).
I guess in the Lisp / ML / Haskell worlds there were people using pretty printers before that, but they weren’t common for Python/JS/C++/Java/shell etc.
Personally I use clang-format and yapf, and they both do line wrapping, but they are not canonicalizers.
They’re not perfect – I tend to disagree with yapf fairly often. But I just use them both, because it’s better than no formatting!
I think if go fmt had line wrapping, it would be fine, so in that sense I disagree with the grandparent comment.
Line wrapping tends to be where people “disagree” more though. A “bad wrap” may cause programmers to abandon a given formatter :-)
If your only input is some kind of AST, then basically you must auto-wrap.
Otherwise everything would be on the same line. The worst case is if you have an S-expression - the whole program would be one line :-)
If you’re fixing the text with an AST, then you don’t have to auto-wrap. You can just use what was there.
So whether line wrapping occurs is related to whether the formatter is a “canonicalizer”, but it’s not identical.
Again clang-format and yapf do line wrapping, but they are not canonicalizers. (and clang-format is the best formatter I’ve used)
Now that I think about it, similar to the AST/CST/lossless syntax tree discussion below, I think some different terms would help. I think “pretty printer” and “formatter” are overloaded.
I think of Wadler’s algorithm not as “pretty printing”, but “line wrapping”. You set up constraints with the groups, compute the width of leaf nodes, and give it a max width. And then it outputs a wrapped program.
But there are other aspects of formatting that are separate, like aligning comments to the same column, or sorting imports alphabetically, which clang-format does.
And also you might want to apply line wrapping separately to code and comments! The comments would have a slightly different algorithm, so line wrapping is more like a “subroutine”
// format formats the given package file originally obtained from src
// and adjusts the result based on the original source via sourceAdj
// and indentAdj.
func format(
fset *token.FileSet,
file *ast.File,
sourceAdj func(src []byte, indent int) []byte,
indentAdj int,
src []byte,
cfg printer.Config,
) ([]byte, error) {
(B) have developers pick the line breaks, like gofmt has them do.
While I’ve never tried gofmt, I mislike how ambiguous this can be. The reason I use formatters is because they enforce consistency throughout a code base, which makes it easier to read and avoids needless bikeshedding across multiple authors.
To anyone who has used gofmt, have you run into issues where the column width is inconsistent when collaborating on code? Is it a real problem, or does it never come up in practice?
Yes! I’ve been meaning to write a blog post about this. I also recently heard Conor on the Tacit Talk podcast (I think it was episode 19) talk about “formatter driven development” where he rearranges Rust code and renames variables in order to get better formatting. I find myself doing this a lot too — it’s wasting the time that formatters are meant to save, but I can’t help it. When adding a character causes rustfmt to explode 1 line into 8, I’ll add another intermediate variable just to bring it back down to 2 lines.
My hot take is column width should be enforced after the initial indentation with the goal of aiding reading. Humans prefer reading around 65 chars per line (which is why book are book-sized), but where that line starts shouldn’t matter.
The recommendation of about 65 characters per line (or 45–90 characters according to another source) was probably based on studies of humans reading prose. But code, unlike prose, is much less often read thoroughly from top to bottom. It is often important to be able to skim code and quickly identify which part of a file is relevant, or to be able to look back and forth between two distant lines to compare them.
Because of these extra requirements, the 65-character guideline might not apply to code. Fitting more characters on each line will reduce the need to scroll to see more code, and though it will be at the expense of the ease of reading the code from top to bottom, that might be a worthwhile tradeoff.
You can still do 80 chars post indentation & it would be readable. What’s bad is when you need to indent a couple of times & now the righthand side is cramped—& I disagree that this should always be seen as some code smell. I find this just a frustrating as very long lines. And I do not think prematurely creating new variable names or methods is the solution. Further, some weirdos will use this as some arbitrary excuse to prevent supporting tabs since that accessibility to allow moving the indentation would stop calculating some exact count.
Is this the first big Go project at Microsoft? I had no idea they could choose Go, I would assume that it just wouldn’t be acceptable, because they wouldn’t want to unnecessarily legitimize a Google project.
They had already lost the web platform by the time they switched to Chromium.
When I think of MS and ‘developers developers developers develop’, I was lead to believe that they’d have more pride in their own development platforms.
I wonder if the switch to Chromium, the embrace of Linux in Azure, the move from VS to VSCode, any of these, would have happened under Ballmer or Gates. I suspect Gates’ new book is like Merkel’s: retrospective but avoiding the most controversial questions: Russian gas and giving up dogfooding. Am I foolish to expect a more critical introspection?
I think corporate open source is now “proven”, and companies don’t worry so much about this kind of thing.
Google has used TypeScript for quite awhile now (even though they had their own Closure compiler, which was not staffed consistently, internally)
All of these companies are best thought of as “frenemies” :) They cooperate in some ways, and compete in others.
They have many common interests, and collaborate on say lobbying the US govt
When Google was small, and Microsoft was big, the story was different. They even had different employee bases.
But now they are basically peers, and employees go freely back and forth
I can see a blog post saying they made changes for FIPS compliance, which is usually understood to be for satisfying government requirements and not a security improvement. I can’t see a summary of any other changes.
MS is a pretty big company, so I wouldn’t be surprised if they have a ton of internal services in Go. Plus with Azure being a big part of their business, I doubt they care what language people use as long as you deploy it on their platforms.
Can confirm that some internal services were written in go as of a couple years ago. (I worked for Microsoft at the time.) I didn’t have a wide enough view to know if it was “a ton,” though.
I’m too lazy to look up a source but some of the error messages I’ve seen in azure make it clear Microsoft uses go to build their cloud.
Message like this one:
dial tcp 192.168.1.100:3000: connect: timeout
Keep in mind Microsoft also maintains their own go fork for fips compliance. I don’t know exactly their use case but I can tell you at Google we did the same thing for the same reason. I think the main fips compliant go binaries being shipped were probably gke/k8s related.
(Edit removed off topic ramble, sorry I have ADHD)
You’re right. I might have ran across this in a kubernetes context since it was 6-7 years ago. I did a quick search just barely and all I saw other than kubernetes was that some azure binaries like azcopy use golang.
The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
(dev lead of TypeScript here, hi!)
We definitely knew when choosing Go that there were going to be people questioning why we didn’t choose Rust. It’s a good question because Rust is an excellent language, and barring other constraints, is a strong first choice when writing new native code.
Portability (i.e. the ability to make a new codebase that is algorithmically similar to the current one) was always a key constraint here as we thought about how to do this. We tried tons of approaches to get to a representation that would have made that port approach tractable in Rust, but all of them either had unacceptable trade-offs (perf, ergonomics, etc.) or devolved in to “write your own GC”-style strategies. Some of them came close, but often required dropping into lots of unsafe code, and there just didn’t seem to be many combinations of primitives in Rust that allow for an ergonomic port of JavaScript code (which is pretty unsurprising when phrased that way - most languages don’t prioritize making it easy to port from JavaScript/TypeScript!).
In the end we had two options - do a complete from-scrach rewrite in Rust, which could take years and yield an incompatible version of TypeScript that no one could actually use, or just do a port in Go and get something usable in a year or so and have something that’s extremely compatible in terms of semantics and extremely competitive in terms of performance.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions). We still want a highly-separated API surface to keep our implementation options open, so Go’s interop shortcomings aren’t particularly relevant. Go has excellent code generation and excellent data representation, just like Rust. Go has excellent concurrency primitives, just like Rust. Single-core performance is within the margin of error. And while there might be a few performance wins to be had by using unsafe code in Go, we have gotten excellent performance and memory usage without using any unsafe primitives.
In our opinion, Rust succeeds wildly at its design goals, but “is straightforward to port to Rust from this particular JavaScript codebase” is very rationally not one of its design goals. It’s not one of Go’s either, but in our case given the way we’ve written the code so far, it does turn out to be pretty good at it.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions)
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Now that I’ve worked on a garbage collector, I see a sweet spot in languages like Go and C# – they have both value types deallocated on the stack and GC. Both Java and Python lack this semantic, so the GCs have to do more work, and the programmer has less control.
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a Foo* type as well as a Foo type, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.
I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between struct (value) and mutable struct (reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.
I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
$ go build -gcflags "-m" main.go
.\main.go:8:14: *y escapes to heap
.\main.go:11:13: x does not escape
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the node.parent reference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forward
Hejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with Object.freeze it isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have a node.parent reference.
So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the node.parent reference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.
The bad part is that then all the nodes you didn’t replace have chains of node.parent references that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutate node.parent, which means that 1) you can’t actually Object.freeze(node) and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.
See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the node.parent reference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.
Hm yeah it was a very good talk. My summary of the type checking part is
The input to the type checker is immutable ASTs
That is, parsing is “embarassingly parallel”, and done per file
They currently divide the program into 4 parts (e.g. 100 files turns into 4 groups of 25 files), and they do what I’d call “soft sharding”.
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess you have to de-duplicate the type errors? Because some type errors might come twice, since you are duplicating some work
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Under 8 cores and 8 threads, the parallel front end can reduce the clean build (cargo build with -Z threads=8 option) time by about 30% on average. (These crates are from compiler-benchmarks of rustc-perf)
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
I guess this is just because, empirically, you don’t get more than 3x speedup.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
Acknowledging some weak spots, Go’s in-proc JS interop story is not as good as some of its alternatives. We have upcoming plans to mitigate this, and are committed to offering a performant and ergonomic JS API.
the alternative (without this work) probably really was 10-15% slower, for builds using clang-19 or newer. For instance, Simon Willison reproduced the 10% speedup “in the wild,” as compared to Python 3.13, using builds from python-build-standalone.
Hm though as far as I know, almost 100% of Linux distros use GCC … So say on Debian and Red Hat, Python 3.14 will only give a 2% improvement?
It seems like that will be noticed widely when those distros update … e.g. if people are expecting 10-15% improvement with a new Python version, but only get 2% :-(
(and btw this is a great investigation, and writeup!)
an example of out-side-of-knowledge thinking –or paradigm shift– which is essentially making the progress of science.
I’m not convinced that concept of sort of spontaneous knowledge, or what sounds as almost a non-causal knowledge, is even possible but it’s interesting - maybe some sort of random selection process + experimentation would play a part here, but that seems perfectly fine to encode into a model. In reality for whatever reason I suspect that all of these people simply processed information and experiences based on biases but ultimately in a way that’s straightforwardly causal, just as any llm would be, and just as I imagine all consciousness is.
These are exactly the kinds of exams where I excelled in my field. These benchmarks test if AI models can find the right answers to a set of questions we already know the answer to.
Well they aren’t known to the AI. The point is to see if the AI can use the building blocks it has to assemble novel solutions (from the AI’s perspective) to new problems. This is distinctly different from school, as described in the article, where you learn the questions and answers at the same time by reading a book on the topic. School rarely gives you X and Y and then tells you to derive Z based on the shared properties of X and Y + some leap to new properties of Z.
If we want scientific breakthroughs, we should probably explore how we’re currently measuring the performance of AI models and move to a measure of knowledge and reasoning able to test if scientific AI models can for instance:
Is this not exactly what those math ones do? The point is that the AI should be able to derive answers to math problems that it hasn’t seen before based on properties of what it has seen before. It seems that these recommendations are largely where things are headed with regards to chain of thought and the benchmarks being used.
PS: You might be wondering what such a benchmark could look like. Evaluating it could involve testing a model on some recent discovery it should not know yet (a modern equivalent of special relativity) and explore how the model might start asking the right questions on a topic it has no exposure to the answers or conceptual framework of.
If we’re just saying “test it on things that are not a part of its training set” we do that already. If it’s “test it on things it can have no conceptual framework of” I don’t believe that humans are capable of solving those problems and I personally believe that it is literally impossible to do so. There has to be some sort of pre-knowledge from which other knowledge can be derived and likely some sort of external force that provides some kind of epistemic access, I suppose the idea here is to minimize those pre-requisites.
I guess my feelings here are that we are already doing what is being advocated for here, mostly?
I think the counterargument is basically “Poincare basically did or would have invented relativity” and such …
Or someone else would have, in a matter of time
I think that’s basically impossible to against. Nobody would say that in 2025, we wouldn’t know about relativity if Einstein died early
Also, I replace “LLM” with “search engines” again. I would say search engines probably reduced some creative thinking in the general case, but for the most creative thinkers, it probably didn’t (?). Or at least I’ve never heard anyone argue that – I’d have to think about it [1]
It’s funny that the “AI” framing confuses both the advocates and the detractors … I more agree with the framing of “pretty reliable word calculators”, and the surprising thing is that “word calculation” can produce some knowledge/insight
(On the other hand, I guess it might be surprising if the best “word calculator” could produce no insight at all!)
[1] Maybe I need to re-read this 2008 article: Is Google Making Us Stupid? What the Internet is doing to our brains
Maybe we are more stupid :) I guess by now we would have seen all the non-search-engine using, non-stupid people achieving more, not sure … My guess is that using ONLY google is a loss, but using google to find primary sources is a HUGE win.
The main mistake people usually make is thinking Newton or Einstein were just scaled-up good students, that a genius comes to life when you linearly extrapolate a top-10% student
I think this is right, but as an exercise I always replace “AI” or “LLM” with “search engine” [1]
Is there any “current” genius that doesn’t use a search engine? I doubt it
Would they be hobbled if they didn’t? Yes, because I guess the alternative is looking through the stacks at the library
Similarly, I think the future geniuses will use LLMs, but they will know how to use them better than we currently do … (i.e. not be fooled so easily)
So in that sense, if you are taking the strict counterfactual of no LLMs, there WILL be a “compressed 21st century”
But the irony is that we won’t think of it as such … Because we ALREADY had a compressed “first 25 years” due to Internet/search engines, but we don’t think of it that way. Mostly we are still coming to grips with what the Internet has done to society! Or at least some people are
(That said, I also didn’t really agree with most of the stuff in the referenced article by the CEO of Anthropic)
So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!
Go back and re-read that sentence. One ampersand!
That is funny … must have been a very big vector !!
And yeah C++ is too subtle here. I guess Go sorta has the same thing, because you can do:
x := bigData
x = &bigData
without declaring the type
In C++ that’s
auto x = bigData;
auto& x = bigData;
But with Go you usually use slices … maybe the real issue is that std::vector is a type with a non-trivial copy constructor.
I’ve come to the conclusion that I’ll not devote time to learning any other version control system, probably ever.
I just don’t have the same issues with git that some people seem to have. I learned the git model long ago and I can get out of any real trouble without really thinking about it. The command line experience for me is just muscle memory at this point, I don’t really care that the commands could be streamlined.
I gave Jujutsu a good once over, read @steveklabnik tutorial and everything, played with it for a bit, but after some time I was just left wondering “why am I doing this?”. It’d just take me much longer to learn JJ than any time it’d save me for the rest of my life.
Maybe I’ll be forced to eat my words one day. I applaud the idea of innovating in this space, however. It seems the newer generation has plenty of trouble with git, something that helps them is useful.
I think git’s probably the local optimum for the next 10-20 years, so I think you’re making a safe bet.
All these experiments suggest to me that people are feeling genuine friction with git though. I still think a “theory of patches” (perhaps related to the one from Darcs) could motivate a new version control system that permits new uses, enables users to get further with less work and fear, and maybe avoids most of the pitfalls. I don’t think we’re there yet though.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface. My own experience of git is like yours. It isn’t easy, but I’m used to it. The replacement will have to make a material improvement day-to-day, larger than a nicer interface. It is, today, really hard for me to imagine feeling the kind of leap that was subversion to git (or mercurial) emerging from a git-to-new-porcelain transition.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface.
As I’ve said here multiple times here and elsewhere, I’m a git fanboy. I personally don’t understand any of these efforts to revamp git’s UI. But I’m a grumpy old man who thinks craftspeople should learn to master their tools when these tools are ubiquitous and good enough. So don’t listen to me :) .
However, even though I agree that the git internals are amazing and well designed, I have to object to the sophism “These experiments also tell me that people agree that git has a very good internal model.”
With this type of logic, I could also state “The effort put into the Wine project tells me that people agree that Windows has a very good graphics API and very good binary format.”
I don’t think it’s the case. Whether it is for Wine or for jj and xit. I think all of these projects are targeting compatibility, and trying to prevent flamewars and bikeshedding over which VCS to use for specific project. I believe the idea behind using the git protocol is: you use xit, they use jj, I use git, everybody is happy.
Although I agree with your disagreement to the logic, I personally find Git’s object model to be rather intuitive. The only sound complaint I’ve encountered recently, imo, is that Git is unable to store directories, but as far as my understanding goes that could be represented as an empty tree object, if Git were to be patched to support that representation.
Notwithstanding that, I do believe that the reason jj/xit/got use the same plumbing as Git is about compatibility.
Also, it’s not necessarily an issue with the newer generation. Young-ish programmers that I know are generally comfortable with Git, including me, while it seems like people used to Mercurial/SVN/CVS who have a hard time groking it.
As I said, I’m a git fanboy. I do think that the git object model is great, but I also see issues. One of my biggest issue is that git is incapable to track renames and copies.
If I rename a file and largely modify that file in the same commit, git will think that I deleted a file called foo.txt, and committed a new different file (written from scratch) called bar.txt. If I want to merge modifications of foo.txt on the lines I didn’t touch, I will have merge conflicts.
Copy is another issue. If I copy foo.txt to bar.txt and modify bar.txt a little bit in one commit, the git object model will think that you just created a new file bar.txt from scratch. If you want to merge work that modified foo.txt before you copied it, the changes will not be propagated to bar.txt.
This is a very intentional decision by git, compared to other systems that tracked renames/copies with metadata (svn and hg, I think).
git is content-based, and the metadata is derived, rather than stored.
It is counterintuitive, but I think git’s choice is better, and I thought most VCS experts basically came around to that conclusion (?)
This rename and branch metadata issue was a big argument around 2007 when git was gaining popularity, but it seems to have been mostly settled …
I think one main problem is if you have a branch-heavy workflow, which turned out to be extremely useful in practice, then the most general solution is for renames to be derived from content. It just does a better job in practice.
Another under-rated thing is that git repositories are extremely backward compatible. A “branch” is almost nothing to the git model, just as a “rename” is literally nothing. This makes A LOT of things work more smoothly – the core is more elegant and compatible
If I want to merge modifications of foo.txt on the lines I didn’t touch, I will have merge conflicts.
I don’t personally run into that problem, and I rename all the time … I am not familiar with the details, but my guess is it’s because I commit very often
And also there are like a half a dozen merge strategies in git, and I guess the experts on big projects use those (but I don’t)
IME the alternatives always run into problems too … branches are cheap in git, but expensive in other systems – both in terms of time and cognitive overhead.
i.e. it doesn’t make sense to criticize the object model on theoretical grounds, when the systems that did the theoretically better thing actually behave worse in practice
I am not sure but Linus might have mentioned this at his ~2007 talk on git at Google, which was famously a bit abrasive (I was there):
And again, to change my mind, I would have to see a system that does better in practice … IME it’s a non-issue with git, and git does significantly better than its predecessors.
Most of the time using git mv instead of mv works for me although there are still a bunch of caveats. IMO this is a porcelain issue rather than an object model one
Pretty naturally, people who have used the feature of reliably-commit-attached branches have more issues with Git not allowing syncing the reflog than the people who were told no-real-branches is the only possibility and formed their model across this lack of functionality!
Yeah same here. To me, git just gets a lot of work done. It was a good 20%-50% permanent improvement over svn and hg in my mind, and that’s a huge win compounded over a decade.
I think many issues can be solved by simply committing more often, which I always did with every VCS. And the others issues I hit along ago and have workarounds for. Stack Overflow is invaluable here.
It’s funny to me that I don’t think I would be happy with git without Google/Stack Overflow, which is a sign of deficiency, but it’s still the best thing. (And yes I treat the command names and flags as opaque crap to memorize.)
I already spend too much time faffing with various tools, so for me git and vim are “set and forget”. On the other hand, I seem to have more problems with distros (Ubuntu was sort of ruined IMO, Debian is OK but can use improvement), and browsers (Firefox and Chromium both have issues).
Interesting, I used to think the same way. I considered myself a git power user and had no problems with it at all. I was very productive, comfortable with editing my history and could fix all my coworkers’ problems. Then jj clicked for me and I couldn’t go back.
I was still slightly stuck with git because it’s more performant in huge repos like the kernel. So I frantically searched for ways to get jj’s productivity and power in git. I finally found the config option rebase.rebaseMerges, which is the crucial missing piece to mimic the megamerge workflow of jj. Other config options are obviously essential too, but rebaseMerges is never really talked about online as far as I can tell.
It’s not at all perfect. git is still more manual work and more friction than jj. And some things still annoyingly don’t work at all. (Why can’t git do an octopus merge when there is a conflict?? jj can.)
So git can definitely be usable and productive. But if a power user like me, with a big config polished over years, cannot get the same power that jj gives you out-of-the boxes and with an intuitive CLI… I definitely see jj as the future “default” VCS people will be using and recommending.
While git is certainly cumbersome from time to time, I’ve learned to live with it. But it doesn’t stop to amaze me how many people just put up with GitHub. Pushing a branch, going to web, clicking a button, filling out a form? Why isn’t pushing a branch enough? And then reviewing online in a GUI so slow it takes a second to open a side-panel with the list of affected files, at least on large PRs. This all feels so backwards.
The allocator design space is huge and nuanced, so at a cursory glance I’m taking the strong claims by the blog post and related PRs/issues with a little skepticism. Definitely looking forward to more rigorous comparisons with other widely used/SOTA allocators!
EDIT: Forgot to mention that the code is readable and easily understood even though I don’t write Zig, really recommend at least perusing the source!
I only see 2 workloads there – a synthetic benchmark, and a Zig compiler workload. Allocator performance is VERY multi-dimensional, so you need more workloads to draw any conclusions
e.g. test on say Firefox, Redis, and SPEC benchmarks, like this paper:
and measure both space and time. Different programs have very different allocation patterns
Otherwise I’d just say “outcompetes glibc malloc on the things I tried”, not “outcompetes glibc malloc”.
I wouldn’t say that that Oils has an allocator that beats glibc too, even though here it runs 74.5% of the time as glibc (IOW more than 25% faster), as measured by cachegrind.
Claude AI claims that glibc uses Knuth’s boundary tag coalescing, described in 1973 in TaoCP. The wiki page doesn’t say that (it seems plausible from what I read?), but that is a real thing:
But this is actually separate from the point I was trying to make, which is that we “just” wrote a pool allocator in 124 lines of code (done by Chris Watkins)
And based on knowledge of the workload, we have a dirt simple policy - there are 2 pools, and one is 24 bytes, and one is 48 bytes. That’s a lot faster than glibc for our workload.
So it’s easy to beat glibc on particular workloads
I’ve heard the claim that “there is no such thing as general purpose memory allocation”, and I tend to believe that … malloc/free is a bare bones interface that can be used in a bazillion ways! There are many dimensions
All that said, I’m not arguing this isn’t a good allocator for certain programs (“idiomatic Zig” ?). And I’d bet that glibc was written / tuned in the days when hardware was much different, e.g. cache misses were 10x-100x less expensive, relatively speaking
Thanks for organizing this event! It was a good use of time – the talks were interesting and well paced
I’d personally be biased toward more “experience reports” than “new ideas”, but I think it’s a good format, and maybe others can take some cues for a variety of events in the same format
I wouldn’t have minded some more organized discussion afterward, although I don’t have any ideas on how to make that happen
I’m also glad Zulip worked out, since TBH if it was on Slack or Discord I may not have joined :)
I wouldn’t have minded some more organized discussion afterward, although I don’t have any ideas on how to make that happen
Yeah, it’s a weird common knowledge problem. There could have been a big discussion the day afterwards if everyone believed there was going to be one. I have thought about spreading the talks out more to give the discussion more room to breathe. Maybe one block per evening or similar.
My intent with “X as Code” was always to get knowledge out of people’s heads and into a more inscribed system. Once inscribed, knowledge and process can be shared, versioned, iterated upon, etc.
This has also been my understanding of X as Code for the better part of a decade - it wasn’t until this year that I realized there are people who understood this differently and disagreed with this definition as passionately as they do.
Interesting! I’ve never cared to wade into it, but I always read “config as code” to mean turing complete as opposed to declarative. Another score for using ambiguous terms.
I’ll add a bit of substance for anyone not convinced. For context, I’m a programmer that knows things like assembly, C, OCaml, Rust and what have you. I can promise you I have done Terraform programming using its more advanced features.
Terraform/HCL has:
modules which are essentially user-defined functions: they have inputs and outputs and can compute anything in between
Anyways all this is besides the point, using code/programming in a way that excludes things that don’t look like your favorite language is unnecessary gatekeeping.
“Code” doesn’t necessarily mean “programming language.” I see code as a superset of programming and configuration languages (and other things like spreadsheets or small plastic bricks).
And the programming and config lang sets have some overlap because, as I said in the other thread, configuration is semantics and can be done in basically any language. For instance, Lua is not always configuration, but with Neovim it mostly is.
I think Mitchell’s aim was to get a description of the system
written down
in a formal notation
in version control
But there are things that can be programmed which cannot be programmed in this kind of code. Pegboard programming of synthesizers or the ENIAC, visual programming like Labview, mechanisms in Lego or Meccano.
There are other non-programming activities that can be turned into code, such as writing (are you using a word processor? or LaTeX?) or diagramming (are you using Inkscape? or Graphviz?).
Is this sentiment a consequence of people misusing “coding” and “coder” to mean programming and programmer?
i.e. “all programming is coding” transforms into “all coding is programming” (not to say that declarative code isn’t programming).
That word choice has always bothered me, it missed the point of what programming’s about.
It looks almost the same as HCL (although I think this was convergent evolution, since I’ve actually never used Terraform):
# this is YSH code!
echo 'hello world'
Data aws_route53_zone cetacean_club {
name = 'cetacean.club.'
}
Resource aws_route53_record A {
zone_id = data.aws_route53_zone.cetacean_club.zone_id
name = "ingressd.$[data.aws_route53_zone.cetacean_club.name]"
type = 'A'
ttl = "300"
}
It can be serialized to JSON, or post-processed and then serialized
Then I show you can wrap Resource in a for loop, as well as parameterize it with a “proc” (procedure).
make-resource (12)
make-resource (34)
if (true) {
make-resource (500)
}
This is all still in progress, and can use feedback, e.g. on Github. (This demo runs, but it relies on a recent bug fix.)
The idea is not really to make something like Terraform, but rather to make a language with metaprogramming powerful enough to make your own “dialects”, like Terraform.
At that time, Oils was a slow Python prototype, but now it’s fast C++! So it’s getting there
The idea of Oils is shell+Python+JSON+YAML, squished together in the same language. So this works by reflection and function calls, not generating text (“Unix sludge”). No Go templates generating YAML, etc.
What ever happened to ‘arc’?
HN recently ported it to run on Common Lisp, so now they can show thousands of comments on the same page etc.
Hm where did you hear that? I’d be interested in hearing details
Do you mean SBCL? As far as I know that is the fast Common Lisp implementation, with a JIT
https://news.ycombinator.com/item?id=41683969
The new Arc implementation, Clarc, is in Common Lisp, so it can use SBCL, which I wouldn’t call a JIT (there are occasional debates.) Clamp was a similar Arc->CL portlet in the past.
Last I checked, SBCL doesn’t do JIT compilation.
I agree, it also seems to be applied rather haphazardly, and generates bad feelings.
I think if content is “self promo”, it just won’t be upvoted by readers – it’s not clear it needs to be a special case rule
(maybe controversial – but I would not mind a downvote button for both stories and comments. You could downvote stories that you think are self promo, as well as accusations of self promo that are “tired” … i.e. it feels like the voting mechanism can solve this problem, without a special case, and without a lot of distracting discussion. The downside is that it can generate “why are people downvoting me?” discussions :-/ )
The downvote feature is a proven abuse magnet. Our flag system is designed to limit that, and it still gets abused.
Where was that proven? Can the mods weigh in on any abuse of flags?
Personally I think the biggest flag missing is “low effort”, like just repeating a joke you can read on Reddit, low effort opinions on Go and Rust, dismissing the article based on the first paragraph, etc.
I’d also be in favor of a “self promo” flag, if that can reduce tedious back and forth about what is self promo
Looks like this was discussed last year: Should we have a “low content” or “low effort” flag?
You can use “me-too” for that kind of comment.
For nix there is:
Am I missing anything? It really feels like that XKCD comic out there.
Probably more like:
Tvix- now renamed to snixDevbox- a wrapper around nix, currently using cppnixIn practice:
The first three are separate for social reasons mostly, the others exist because they want to take a different direction in the actual software.
Lix also has one extra experiment: pipe-operator since v2.91 (August 2024).
I love that feature dearly and use it in my personal config to avoid deeply nested expressions:
h z (g y (f x)) == f x |> g y |> h zlib.pipe (f x) [ (g y) (h z) ]Not saying a pipe operator is bad, but we have an option now with ‘good enough’ ergonomics. Does the pipe operator offer some performance gain by being in the language?
Not sure about performance, but having just written a bunch of Nix code using
|>, I much prefer the operator overlib.pipefor aesthetical reasons:I don’t even want to translate it to
lib.pipe, it would not be much of an improvement over the original, pipe/operator-less form. I do not findlib.pipe’s ergonomics “good enough”.I used
lib.pipebefore enabling the experiment, and I like that function too! I just like the pipe syntax even more.I’m using it in my personal config, which no-one else ever uses/sees, just for the syntax. And
lib.pipeis the obvious, and easy to switch to, off ramp if I need it in the future.Performance is one of the RFC’s unresolved questions and I haven’t noticed a difference because most of my config eval time is not spent in
lib.pipe.I don’t want to duplicate the whole RFC or its discussion here, but having the functionality in the language has other UX benefits like better tracebacks.
I hadn’t seen the RFC.
This is precisely the advantage I could see if it were a built-in—otherwise the value add is pretty slim to keeping it simple just sticking with
lib.pipe(& I say that having used pipe-like operators in PureScript, Elm, OCaml, Haskell, & Nickel and a fan of operators). Recently tho I’ve been coming to the opinion that our function notation sucks & should be post-fix by default ala reverse Polish notation.IIRC the Guix daemon is a fork of the cppnix daemon
Not nix, but similar/related: https://brioche.dev/
If we’re naming projects that don’t implement Nix but are vaguely spiritually related, I think zilch deserves a mention as well.
https://dagger.io/
Guix is an unrelated implementation of the same idea as Nix, using Scheme as the configuration language instead of Nix.
Guix started as a Nix fork, so I wouldn’t say it’s an unrelated implementation
Guix is not a fork of Nix.
My source is:
https://en.wikipedia.org/wiki/GNU_Guix#History
https://lists.gnu.org/archive/html/gnu-system-discuss/2012-06/msg00003.html
I don’t know how much code it shared, but most people seem to say it started as a fork
It did, it took the Nix daemon and modified it. For some time they were still subject to the same CVEs. There’s been an effort to rewrite the daemon in Scheme, but no idea how that’s going (maybe it’s done?).
For some reason a few Guix-affiliated people refuse to acknowledge that they forked the Nix daemon, it’s probably something political.
The nix daemon is a very small piece of the overall system. That piece was borrowed, indeed, but 99% of the system is not that. It is an otherwise completely separate implementation of the functional package management paradigm that makes some radically different design choices. The daemon thing is a distraction. Guix is not a fork of Nix in any way that matters.
Wait until you hear about Linux distros…
I did hear about Linux distros. I think the distro situation has squandered thousands of hours of human effort over pointless minute differences and archaic 90s software design choices.
Is it bad to have many implementations? Genuine question, I don’t know. People also attack Rust for having only 1 serious implementation.
The politics are a little sour here and there, which is probably the biggest downside. Otherwise, good point. The Lix and Snix ports seems really healthy and forward looking. Maybe it causes some confusion for non-NixOS users? But in NixOS, most probably use whatever they are fed by the base system.
I think the danger is that they might start diverging on Nix-the language and we get incompatible Nix repositories. This is already the case to some extend with e.g. flakes (though cppnix/lix/determinate all support them), but could get worse if implementations start adding implementation-specific features.
I think this risk is real, and is currently mitigated primarily by one thing: The steadfastness of nixpkgs maintainers in refusing to accept code in the packages that doesn’t work in Nix 2.3 - which is ancient at this point.
What I can see happening is a sort of Cambrian explosion of Nix implementations and dialects, but with a shared minimal viable subset in “whatever is needed to use nixpkgs”. Sort of like the situation with C, where a bunch of compilers all implement the ISO/IEC standard but also offer a ton of nonstandard extensions, with nixpkgs acting as somewhat of an ad-hoc standards body.
Maybe a flowchart is in order
I generally agree with the 4 tier categorization:
But I’d also add 2 or 3 more tiers:
The goal of YSH is actually to unify tiers 4, 5, 6, and 7 under one language. The slogan I’ve been using is “minimal YSH is shell+Python+JSON+YAML”
Instead of having Unix sludge (autotools - m4 generating make) and Cloud sludge (Helm - Go templates generating YAML), you have one language
This is a hard design challenge, but I just made a release with an overhaul of Hay - https://oils.pub/release/0.28.0/
Hay version 1 was hard-coded in the interpreter - https://oils.pub/release/0.28.0/doc/hay.html
But we realized it’s actually better to self-host it in YSH, using YSH reflection. We will be testing this by rewriting Hay in YSH
So that’s our language design response to https://news.ycombinator.com/item?id=43386115
Instead of tiers 4, 5, 6 being silo’d, we have them all under YSH and the Oils runtime (which is tiny, 2.3 MB of pure native code).
(As a bonus, OSH also runs on the Oils runtime, and it’s the most bash-compatible shell!)
[1] Garbage Collection Makes YSH Different - https://www.oilshell.org/blog/2024/09/gc.html
Shell, Awk, and Make Should Be Combined - https://www.oilshell.org/blog/2016/11/13.html - all these languages lack GC!
[2] Survey of Config Languages - https://github.com/oils-for-unix/oils/wiki/Survey-of-Config-Languages - divides this category into 5 tiers:
[3] Zest: Notation and Representation addresses this - https://www.scattered-thoughts.net/writing/notation-and-representation/
YSH also has a common subset with J8 Notation (which is a superset of JSON)
This is really apt, I like your characterization of it :-)
Similarly, you can implement Ruby-like DSLs in YSH – e.g. Ruby has “Rake” for “Make”: https://github.com/ruby/rake/blob/master/doc/rational.rdoc
IMO it would be nicer in YSH:
You also don’t need to “shell out”, because YSH is already a shell! Both traditional Make and Ruby Rake delegate to the shell, which makes escaping non-obvious, among other things
So if anyone wants to help prove our reflection APIs in this way, feel free to join https://oilshell.zulipchat.com/
Again, we prefer using reflection on data structures in the runtime, rather than “text sludge”
Ruby is my favorite language, but this looks super neat!
Thanks for noticing! I would definitely like more Ruby users to check out YSH (feel free to join https://oilshell.zulipchat.com/ or send other interested people that way)
A slogan is “YSH is for Python and JavaScript users who avoid shell” … But that really includes all of Python/JS/Ruby/Perl/PHP/Lua (though Ruby and Perl users don’t seem to avoid shell as much, which I think is good!)
Partly this is because I know Python and JS pretty well, but not Ruby. But I’m interested in and respect Ruby, e.g. I generated some “survey code” as part of the YSH language design, e.g.
https://github.com/oils-for-unix/oils/blob/master/demo/survey-closure.rb
I think our reflection compares favorably to Ruby, e.g. instead of
bindingwe haveI wrote some design notes in June 2023 about this:
There I wrote
Which is almost the same thing!
But yeah I hope you can do almost anything with YSH that you can do with Ruby, though we need to test it.
One difference is that we don’t have declared params to the
{ }blocks, while Ruby hasBut so far it seems OK to have
xandybe “implicit”, rather than explicit (?) Feedback is welcomeAnother tidbit I remember is a Gary Bernhardt talk, where he says that the thing that Ruby has, that Python doesn’t, is blocks
e.g. for the RSpec test framework and so forth – it makes it nicer
And I agree!
So yeah YSH is influenced by Python and JS, but it has Ruby-like blocks too! I think that adds a lot …
I have been missing blocks for all these years in Python-land :)
The C source for this Thompson v6 shell is quite archaic, using
=+instead of+=and doing a very funky syntax tree representation. Understandable since C itself was evolving (being evolved by the same person, actually), but has anyone ported it to something more modern like ANSI C 89?Tsh by the same author would appear to be that
I agree, though that link you have (same guy/website as TFA, actually) only links to enhanced tsh / etsh or “osh” sources that I can see. The very oldest release from 2003, osh-030730.tar.gz, is starting to look closer to the structure of the v6 source version, but still has a lot more distance than I was thinking a “port” would have. Anyway, THANK YOU!
xv6 has a shell in modern C:
https://github.com/mit-pdos/xv6-public/blob/master/sh.c#L100
I like to look at how they implement pipelines, which is:
and then let’s look at Thompson shell - https://v6sh.org/src/sh.c
So they both do 2 recursive invocations – to runcmd() or execute(), on the left and right node of the AST.
Oh weird, but actually now I notice a difference … xv6 is less efficient?
pipe()and 2fork()calls for every PIPE nodeSo basically in xv6:
is executed like:
That is, you would expect there to be 4
catprocesses to run, but there are at least 2 more processes.But not Thompson shell? I’m curious if I’m reading that right …
BTW https://oils.pub/ stores all the pipeline components in a list, so it doesn’t have this left/right recursion. I’m pretty sure most shells, like bash and dash, are like that – you can have a variable number of children in an AST node, not just 2
The xv6 shell also has a different syntax - no backslash escaping or any quoting and no
$varsand probably other differences. It is also very different in structure and function, as you observe. It is more a re-implementation of an “even more simple” shell-like thing than even a “loose following” of the v6 shell. I’m unsure how many similar statements might apply to other aspects of the xv6 project.EDIT: btw, yes you are reading it correctly. I patched & compiled the xv6 sh.c to just run on Linux and then ran it as ./a.out and then just copy pasted cat|cat|cat|cat with the programs waiting on the terminal and
ps(well, really mypd) output looks like this:Oh cool, thanks for checking! I remember noticing that quirk with xv6, and yeah it’s good to know it wasn’t present in the original Thompson shell
I think we’re splitting hairs now but AFAIU vars and backslash only came with the PWB shell?
If you read the OG v6 shell source in TFA, you can see these things handled (at least $0 $1 and $$). { Might well all be just academic/historical/hair splitting, but that’s sort of the whole topic, too. :-) }
Through the whole thing a huge [citation needed] was glaring so bright I barely could finish it.
[citation needed]
GitHub, the very site it was published on is written in Ruby. By the OP classification it’s a level 4 language. As of right now GitHub is 17 years old. GitHub is also 6,500 strong company. Granted, not everyone is working on GitHub but even if it’s only 20% it’s still 1,300 people working on a Ruby project. they seem to manage, more or less.
Shopify is another big Ruby app. Founded 2 years before GitHub (so 19 years old). At this time about 8,300 people work there. By the same 20% assumption, it’s 1,600 people working on a Ruby app. The seem to manage even better than GitHub.
These are not the only two examples out there. It’s the two I think most of you are familiar.
So when does eventually come? Or at what point the team is too big?
[citation needed]
I would wager a solid $1 that JS, Python, and PHP alone employ more developers than every other language combined. I’m also fairly confident that code in those languages is produced at higher rate, as well.
[citation needed]
Rails (the thing GitHub and Shopify are written on) is arguably a skyscraper of meta-programming. Every Rails-like framework employs meta-programming to some degree. It wades a bit into the weeds of semantics but Spring (Java framework) annotations are a kind of met-programming, so are Rust
#[derive(...)]and other annotations, even C pre-processor. Meta-programming is all over the place and was since the dawn of programming. It’s in every production system even if it doesn’t support eval at runtime.[citation needed]
I’d argue that different syntaxes at different levels is more beneficial as it helps switching context. You’re in for a whole lot of… adventures when you think you’re writing one language when in fact you’re writing a completely different one.
Take this “RustScript” for example:
There’s a whole lot of questions I have about this line. How do you map this onto a lower level Rust? You have to map it onto the lower level somehow because it’s gotta be “effortless”. But how do you infer a type of this? Is this type equivalent to every struct with those fields? How do we convert between the types if the conversion is not defined in the lower level?
Consider this:
Which type is assigned to each of these variables?
Is this a valid RustScript code?
Can we call lower-level functions with any of these?
Would both of these work? What type would they have?
How dynamic is RustScript? The original line is a valid JS. It creates an object and you can add fields to it to your hears content:
Can you do it in RustScript? It’s supposed to be dynamically typed. What would happen to the type? Can you still use this awesome rect with low-lever Rust-proper?
This is just from a single line in the example. I’m sure there will be much much more questions when expanded to the whole of Rust.
Now, don’t get me wrong, I appreciate the idea. It can be an interesting thing to explore. I’m just a bit miffed that OP is backwards. It makes some dubious claims to set the stage for the idea and does no exploration of it at all. I’d rather it started with the core idea of language sets and explored it in more depth. Instead of stripping down Rust syntax to look like JS I’d love OP think about interop (arguably, the core feature of a language set) and how it might work.
I think Shopify is a Sorbet user; if so their codebase moved from level 4 to level 3. Dunno if Github uses Sorbet.
Re. metaprogramming, the dynamic qualifier is important. I like to write code that writes code, but that’s almost always static codegen. Code that rewrites itself is far too brainbending!
Yeah I think the opposite direction bears just as much fruit :-)
You could start with the most ergonomic/productive thing and make it fast – in contrast to starting with the fast, complex language, and simplifying it.
We did that with https://oils.pub/ – started with Python, and then statically typed it with MyPy, and then translated that to garbage-collected C++
So we went from tier 4 to tier 3 to tier 2 !
bash is written in a tier 1 language – C. And OSH is faster than bash in many cases, although there are still cases where we’re slower.
We are able to be faster because bash is a very suboptimal program! (e.g. it uses a ton of linked lists, not sure there are any hash tables at all) Most big programs are suboptimal, especially big C programs.
Wow! I really want to know how the images were made …
I do wonder about AI, but at least when I tried the AI generators more than a year ago, they couldn’t produce anything like this. They produced stuff with a very distinct style, more like this - https://www.youtube.com/@protocoltownhall/videos - everything there is clearly AI
Many of the images look photo-realistic, but then I feel like this page would have taken like 2 years to make, which seems implausible (?)
Oh well now I see she is working on AI, so I guess it must be AI … that was not my first thought, it seemed almost like the opposite aesthetic - https://wattenberger.com/
Or maybe it is some advanced Photoshop plugin that does style transfer of the “Yarn”, but many of those plugins work with AI techniques
(Also, the plea for interfaces that use our senses of course reminds me of https://dynamicland.org/)
i was very impressed with this page when i first saw it, but i refrained from sharing it here, as the author admits to using AI to generate these images, on bluesky: https://bsky.app/profile/patrickpietens.dev/post/3lk4qz7gg7k2n (on the first reply there)
it becomes more obvious towards the end (upon observing the closeups of each characters’ faces), that there is no similarity from one image to the next (the yarn pattern is of different sizes, the backgrounds are blurry and meaningless etc.)
Well thats dissapointing, but not surprising: Nobody would put effort like that for post like this, especially in tech field.
At least I can now just ignore the post, since there wasn’t any effort to begin with.
With respect, I would disagree on the effort part. Yes, they used AI to generate a bunch of images. Throughout history we have used tools to make us more efficient. It seems to me that this is not all that different from that. They had an idea and wanted an article on it, but might not be that much of an artist OR didn’t want to spend the tens or hundreds of hours it would take to make this.
That is just my take.
I’ve not managed to get these sorts of results myself (nor have I really tried, in fairness), but there was an AI art Turing test that did the rounds a few months back that showed some pretty stunning examples of AI-based art that doesn’t just look like standard art.
I have no idea what the actual process looks like for generating these images, though, and can claim absolutely no expertise in this matter.
EDIT: And just like you, the main thing that made me realise the images were AI generated was the amount of time it would have taken to create them if they were real, although after that point it became easier to notice things that were a little bit off, like odd features or weird blurring. Although maybe those would have also been present in real photos, and I was just primed to notice them because I knew the images were AI?
Interesting article … I think this page looks a little more surprising than those examples, probably because it’s an experienced designer + AI, not “one shot” image generation.
Although I don’t know exactly how all those were generated … but definitely a few of them toward the end have the giveaway “AI style”
And yeah I am judging it based on style, and I know AI can generate multiple styles … but yeah I think this is a good example of augmentation/acceleration, just like you’d see with coding. A random person is not going to get as good results from the LLMs as an experienced programmer!
In this context it’s useful to distinguish between an AST and a concrete syntax tree: a CST includes all the details of spelling, spacing, and commentary that are omitted from an AST.
The article does mention the term “concrete syntax tree”:
But I actually disagree with your usage, and the article’s usage! I would say it’s more like:
1 * (2 + 3), but does NOT contain comments.I made up the “lossless syntax tree” term based on what I knew about Clang and Go: From AST to Lossless Syntax Tree (2017, wow 8 years ago now)
Go was the language that popularized auto-formatting, and Clang came quickly after.
So you can quibble with my terminology, but if you Google it, it’s reasonably popular, though by no means standard.
Wiki page - https://github.com/oils-for-unix/oils/wiki/Lossless-Syntax-Tree-Pattern
All these sources agree that a CST is a synonym for “parse tree”, and contains the derivation:
https://en.wikipedia.org/wiki/Parse_tree
Abstract vs. Concrete Syntax Trees - https://eli.thegreenplace.net/2009/02/16/abstract-vs-concrete-syntax-trees/
What’s the difference between parse trees and abstract syntax trees (ASTs)? - great diagram, and cites a book “Compilers and Compiler Generators”
I checked the terminology of the Dragon Book specifically for this comment:
So the Dragon Book is making what I call the CST vs AST distinction, with slightly different terms.
Additionally, I make a “lossless syntax tree” distinction.
The key difference is that the Dragon book only talks about compiling, not formatting or translating. I think “lossless syntax tree” is useful for talking about what Clang and Go do – they are neither CSTs nor ASTs.
Their trees don’t “recapitulate” the grammar – quite the contrary, their parsers are hand-written. But they contain all the physical details of the code – not just parens, but also whitespace and comments.
You can fight me, but this is exactly what makes gofmt excellent, and rustfmt very annoying. zig fmt is best though, there, the user controls line breaks via stuff like trailing commas.
I see from the upvotes there’s a perspective here, and I’d like to understand it.
For contrast, here’s my perspective, from which
rustfmtis good andgofmtis bad.rustfmtpick the line breaks, or (B) have developers pick the line breaks, likegofmthas them do. Option (B) allows artisinal hand-crafted code formatting, with three downsides: (i) it requires more effort by developers that to me feels like busywork, I’d rather focus on the code itself, (ii) it’s going to be less consistent than what an automatic code formatter gives you, and (iii) realistically not everyone is going to obey any particular line length limit, and there’s going to be soft wrapping.What’s the other perspective?
rustfmt routinely produces suboptimal formatting, where things that should be multilined are crammed into a single line, and things that should be compact are unnecessarily spread out. A related problem is that formatting is self-inconsistent: whether something is multiple or single line depends on an arbitrary threshold, and not on the formatting of three logically related expressions right before. Given that the code is read way more often than written, it is a good trade to spend a little more time writing it, to make it more readable, as long as human only chooses formatting, with the tool doing the actual job of adjusting whitespace. Putting just a little mind into formatting is especially useful because often the best way to format a long expression is to break it apart.
The tradeoff here of course depends on a particular team, as choosing a tasteful way to present your code requires taste!
You still of course want to enforce line length in CI.
I suspect a lot of the difference in perspective is that my experience involves a mix of:
rustfmt: learning to expect what it does, and coming to value its mechanistic predictability. For example, knowing that if function arguments are split across multiple lines then there’s exactly one per line makes a lot of things easier (e.g. comparing args across functions, re-ordering args).Having co-workers who present code tastefully sounds nice too.
Good point about enforcing line length in CI, that completely mitigates my point (iii).
I am not sure, but I think the original comment might be complaining more about “over-canonicalization”, which we discussed last year.
So I think this part of the article is wrong:
It is leaving out the go fmt / clang-format / yapf / scala fmt type of formatter, which we also discussed. They do not re-print the syntax tree; they use some kind of tree to fix the text.
And I’d claim that
go fmtis the tool that made formatting “universal” among working programmers (although I don’t use Go myself, so I haven’t really used it).I guess in the Lisp / ML / Haskell worlds there were people using pretty printers before that, but they weren’t common for Python/JS/C++/Java/shell etc.
Personally I use clang-format and yapf, and they both do line wrapping, but they are not canonicalizers.
They’re not perfect – I tend to disagree with yapf fairly often. But I just use them both, because it’s better than no formatting!
I think if go fmt had line wrapping, it would be fine, so in that sense I disagree with the grandparent comment.
Line wrapping tends to be where people “disagree” more though. A “bad wrap” may cause programmers to abandon a given formatter :-)
Oh 100%. Go made it mainstream. And once you’re in fmt land you can’t go back, so it spread to other languages.
Oh and to be clearer:
So whether line wrapping occurs is related to whether the formatter is a “canonicalizer”, but it’s not identical.
Again clang-format and yapf do line wrapping, but they are not canonicalizers. (and clang-format is the best formatter I’ve used)
Now that I think about it, similar to the AST/CST/lossless syntax tree discussion below, I think some different terms would help. I think “pretty printer” and “formatter” are overloaded.
I think of Wadler’s algorithm not as “pretty printing”, but “line wrapping”. You set up constraints with the groups, compute the width of leaf nodes, and give it a max width. And then it outputs a wrapped program.
But there are other aspects of formatting that are separate, like aligning comments to the same column, or sorting imports alphabetically, which clang-format does.
And also you might want to apply line wrapping separately to code and comments! The comments would have a slightly different algorithm, so line wrapping is more like a “subroutine”
Note to myself, I looked up the
go fmtsource, and it an “adjuster” or “fixer”, not a “tree printer”https://github.com/golang/go/blob/master/src/cmd/gofmt/internal.go#L94
While I’ve never tried
gofmt, I mislike how ambiguous this can be. The reason I use formatters is because they enforce consistency throughout a code base, which makes it easier to read and avoids needless bikeshedding across multiple authors.To anyone who has used
gofmt, have you run into issues where the column width is inconsistent when collaborating on code? Is it a real problem, or does it never come up in practice?Yes! I’ve been meaning to write a blog post about this. I also recently heard Conor on the Tacit Talk podcast (I think it was episode 19) talk about “formatter driven development” where he rearranges Rust code and renames variables in order to get better formatting. I find myself doing this a lot too — it’s wasting the time that formatters are meant to save, but I can’t help it. When adding a character causes rustfmt to explode 1 line into 8, I’ll add another intermediate variable just to bring it back down to 2 lines.
My hot take is column width should be enforced after the initial indentation with the goal of aiding reading. Humans prefer reading around 65 chars per line (which is why book are book-sized), but where that line starts shouldn’t matter.
The recommendation of about 65 characters per line (or 45–90 characters according to another source) was probably based on studies of humans reading prose. But code, unlike prose, is much less often read thoroughly from top to bottom. It is often important to be able to skim code and quickly identify which part of a file is relevant, or to be able to look back and forth between two distant lines to compare them.
Because of these extra requirements, the 65-character guideline might not apply to code. Fitting more characters on each line will reduce the need to scroll to see more code, and though it will be at the expense of the ease of reading the code from top to bottom, that might be a worthwhile tradeoff.
You can still do 80 chars post indentation & it would be readable. What’s bad is when you need to indent a couple of times & now the righthand side is cramped—& I disagree that this should always be seen as some code smell. I find this just a frustrating as very long lines. And I do not think prematurely creating new variable names or methods is the solution. Further, some weirdos will use this as some arbitrary excuse to prevent supporting tabs since that accessibility to allow moving the indentation would stop calculating some exact count.
Is this the first big Go project at Microsoft? I had no idea they could choose Go, I would assume that it just wouldn’t be acceptable, because they wouldn’t want to unnecessarily legitimize a Google project.
Their web browser runs on Chromium, I think it’s a little late to avoid legitimizing Google projects
They had already lost the web platform by the time they switched to Chromium.
When I think of MS and ‘developers developers developers develop’, I was lead to believe that they’d have more pride in their own development platforms.
I wonder if the switch to Chromium, the embrace of Linux in Azure, the move from VS to VSCode, any of these, would have happened under Ballmer or Gates. I suspect Gates’ new book is like Merkel’s: retrospective but avoiding the most controversial questions: Russian gas and giving up dogfooding. Am I foolish to expect a more critical introspection?
I think corporate open source is now “proven”, and companies don’t worry so much about this kind of thing.
Google has used TypeScript for quite awhile now (even though they had their own Closure compiler, which was not staffed consistently, internally)
All of these companies are best thought of as “frenemies” :) They cooperate in some ways, and compete in others.
They have many common interests, and collaborate on say lobbying the US govt
When Google was small, and Microsoft was big, the story was different. They even had different employee bases. But now they are basically peers, and employees go freely back and forth
Microsoft has actually maintained a security-hardened build of Go for a while now: https://devblogs.microsoft.com/go/
I can see a blog post saying they made changes for FIPS compliance, which is usually understood to be for satisfying government requirements and not a security improvement. I can’t see a summary of any other changes.
MS is a pretty big company, so I wouldn’t be surprised if they have a ton of internal services in Go. Plus with Azure being a big part of their business, I doubt they care what language people use as long as you deploy it on their platforms.
Can confirm that some internal services were written in go as of a couple years ago. (I worked for Microsoft at the time.) I didn’t have a wide enough view to know if it was “a ton,” though.
We embrace and we extend!
I’m too lazy to look up a source but some of the error messages I’ve seen in azure make it clear Microsoft uses go to build their cloud.
Message like this one:
Keep in mind Microsoft also maintains their own go fork for fips compliance. I don’t know exactly their use case but I can tell you at Google we did the same thing for the same reason. I think the main fips compliant go binaries being shipped were probably gke/k8s related.
(Edit removed off topic ramble, sorry I have ADHD)
Could this message just be coming from Kubernetes?
You’re right. I might have ran across this in a kubernetes context since it was 6-7 years ago. I did a quick search just barely and all I saw other than kubernetes was that some azure binaries like
azcopyuse golang.The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
Source: https://www.reddit.com/r/typescript/comments/1j8s467/comment/mh7ni9g
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
https://github.com/danos/yang/commit/c98b220f6a1da7eaffbefe464fd9e734da553af0
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
That is great explanation of “Why Go and not Rust?”
If you’re looking for “Why Go and not AOT-compiled C#?” see here: https://youtu.be/10qowKUW82U?t=1154s
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
That interview is really interesting, worth watching the whole thing.
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Related comment about GC and systems languages (e.g. once you have a million lines of C++, you probably want GC): https://lobste.rs/s/gpb0qh/garbage_collection_for_systems#c_rrypks
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
https://lobste.rs/s/2ah6bi/programming_without_pointers#c_5g2nat
[1] BTW Guy Steele’s famous 1998 “growing a language” actually advocated value types in Java. AFAIK as of 2025, “Project Valhalla” has not landed yet
Compilers written in OCaml are famous for being super-fast. See eg OCaml itself, Flow, Haxe, BuckleScript (now ReScript).
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Maybe, but given that Flow, its main competitor, distributes binaries for all major platforms: https://github.com/facebook/flow/releases/tag/v0.264.0
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Also Nim.
Also Julia.
There surely are others.
Yes good points, I left out Nim and Julia. And apparently Crystal - https://colinsblog.net/2023-03-09-values-and-references/
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a
Foo*type as well as aFootype, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between
struct(value) andmutable struct(reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
https://forum.golangbridge.org/t/how-gc-handles-interior-pointer/36195/5
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
Yes, that is a bit of control you give up with Go. Searching for “golang escape analysis”, this article is helpful:
https://medium.com/@trinad536/escape-analysis-in-golang-fc81b78f3550
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Yes, the stack/heap distinction is made by the compiler, not the programmer, in Go.
Why did you leave JS/TS off the list? They seem to have left it off too and that confuses me deeply because it also has everything they need
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the
node.parentreference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forwardHejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
Really, you should watch the interview https://youtu.be/10qowKUW82U
AIUI a typical LSP implementation integrates all the tools you listed so they are sharing a syntax tree already.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with
Object.freezeit isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have anode.parentreference.So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the
node.parentreference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.The bad part is that then all the nodes you didn’t replace have chains of
node.parentreferences that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutatenode.parent, which means that 1) you can’t actuallyObject.freeze(node)and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the
node.parentreference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.You are not joking, just the hack to make type checking itself parallel is well worth an entire hour!
Hm yeah it was a very good talk. My summary of the type checking part is
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
Can you share the timestamp to the discussion of this hack, for those who don’t have one hour?
I think this one: https://www.youtube.com/watch?v=10qowKUW82U&t=2522s
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
They also posted a (slightly?) different response on GitHub: https://github.com/microsoft/typescript-go/discussions/411
Yes please!
Hm though as far as I know, almost 100% of Linux distros use GCC … So say on Debian and Red Hat, Python 3.14 will only give a 2% improvement?
It seems like that will be noticed widely when those distros update … e.g. if people are expecting 10-15% improvement with a new Python version, but only get 2% :-(
(and btw this is a great investigation, and writeup!)
I’m not convinced that concept of sort of spontaneous knowledge, or what sounds as almost a non-causal knowledge, is even possible but it’s interesting - maybe some sort of random selection process + experimentation would play a part here, but that seems perfectly fine to encode into a model. In reality for whatever reason I suspect that all of these people simply processed information and experiences based on biases but ultimately in a way that’s straightforwardly causal, just as any llm would be, and just as I imagine all consciousness is.
Well they aren’t known to the AI. The point is to see if the AI can use the building blocks it has to assemble novel solutions (from the AI’s perspective) to new problems. This is distinctly different from school, as described in the article, where you learn the questions and answers at the same time by reading a book on the topic. School rarely gives you X and Y and then tells you to derive Z based on the shared properties of X and Y + some leap to new properties of Z.
Is this not exactly what those math ones do? The point is that the AI should be able to derive answers to math problems that it hasn’t seen before based on properties of what it has seen before. It seems that these recommendations are largely where things are headed with regards to chain of thought and the benchmarks being used.
If we’re just saying “test it on things that are not a part of its training set” we do that already. If it’s “test it on things it can have no conceptual framework of” I don’t believe that humans are capable of solving those problems and I personally believe that it is literally impossible to do so. There has to be some sort of pre-knowledge from which other knowledge can be derived and likely some sort of external force that provides some kind of epistemic access, I suppose the idea here is to minimize those pre-requisites.
I guess my feelings here are that we are already doing what is being advocated for here, mostly?
I think the counterargument is basically “Poincare basically did or would have invented relativity” and such …
Or someone else would have, in a matter of time
I think that’s basically impossible to against. Nobody would say that in 2025, we wouldn’t know about relativity if Einstein died early
Also, I replace “LLM” with “search engines” again. I would say search engines probably reduced some creative thinking in the general case, but for the most creative thinkers, it probably didn’t (?). Or at least I’ve never heard anyone argue that – I’d have to think about it [1]
It’s funny that the “AI” framing confuses both the advocates and the detractors … I more agree with the framing of “pretty reliable word calculators”, and the surprising thing is that “word calculation” can produce some knowledge/insight
(On the other hand, I guess it might be surprising if the best “word calculator” could produce no insight at all!)
[1] Maybe I need to re-read this 2008 article: Is Google Making Us Stupid? What the Internet is doing to our brains
https://www.theatlantic.com/magazine/archive/2008/07/is-google-making-us-stupid/306868/
Maybe we are more stupid :) I guess by now we would have seen all the non-search-engine using, non-stupid people achieving more, not sure … My guess is that using ONLY google is a loss, but using google to find primary sources is a HUGE win.
I think this is right, but as an exercise I always replace “AI” or “LLM” with “search engine” [1]
Is there any “current” genius that doesn’t use a search engine? I doubt it
Would they be hobbled if they didn’t? Yes, because I guess the alternative is looking through the stacks at the library
Similarly, I think the future geniuses will use LLMs, but they will know how to use them better than we currently do … (i.e. not be fooled so easily)
So in that sense, if you are taking the strict counterfactual of no LLMs, there WILL be a “compressed 21st century”
But the irony is that we won’t think of it as such … Because we ALREADY had a compressed “first 25 years” due to Internet/search engines, but we don’t think of it that way. Mostly we are still coming to grips with what the Internet has done to society! Or at least some people are
(That said, I also didn’t really agree with most of the stuff in the referenced article by the CEO of Anthropic)
[1] like https://news.ycombinator.com/item?id=43013363
That is funny … must have been a very big vector !!
And yeah C++ is too subtle here. I guess Go sorta has the same thing, because you can do:
without declaring the type
In C++ that’s
But with Go you usually use slices … maybe the real issue is that
std::vectoris a type with a non-trivial copy constructor.I’ve come to the conclusion that I’ll not devote time to learning any other version control system, probably ever.
I just don’t have the same issues with git that some people seem to have. I learned the git model long ago and I can get out of any real trouble without really thinking about it. The command line experience for me is just muscle memory at this point, I don’t really care that the commands could be streamlined.
I gave Jujutsu a good once over, read @steveklabnik tutorial and everything, played with it for a bit, but after some time I was just left wondering “why am I doing this?”. It’d just take me much longer to learn JJ than any time it’d save me for the rest of my life.
Maybe I’ll be forced to eat my words one day. I applaud the idea of innovating in this space, however. It seems the newer generation has plenty of trouble with git, something that helps them is useful.
I think git’s probably the local optimum for the next 10-20 years, so I think you’re making a safe bet.
All these experiments suggest to me that people are feeling genuine friction with git though. I still think a “theory of patches” (perhaps related to the one from Darcs) could motivate a new version control system that permits new uses, enables users to get further with less work and fear, and maybe avoids most of the pitfalls. I don’t think we’re there yet though.
These experiments also tell me that people agree that git has a very good internal model and a very clumsy interface. My own experience of git is like yours. It isn’t easy, but I’m used to it. The replacement will have to make a material improvement day-to-day, larger than a nicer interface. It is, today, really hard for me to imagine feeling the kind of leap that was subversion to git (or mercurial) emerging from a git-to-new-porcelain transition.
As I’ve said here multiple times here and elsewhere, I’m a git fanboy. I personally don’t understand any of these efforts to revamp git’s UI. But I’m a grumpy old man who thinks craftspeople should learn to master their tools when these tools are ubiquitous and good enough. So don’t listen to me :) .
However, even though I agree that the git internals are amazing and well designed, I have to object to the sophism “These experiments also tell me that people agree that git has a very good internal model.”
With this type of logic, I could also state “The effort put into the Wine project tells me that people agree that Windows has a very good graphics API and very good binary format.”
I don’t think it’s the case. Whether it is for Wine or for jj and xit. I think all of these projects are targeting compatibility, and trying to prevent flamewars and bikeshedding over which VCS to use for specific project. I believe the idea behind using the git protocol is: you use xit, they use jj, I use git, everybody is happy.
Although I agree with your disagreement to the logic, I personally find Git’s object model to be rather intuitive. The only sound complaint I’ve encountered recently, imo, is that Git is unable to store directories, but as far as my understanding goes that could be represented as an empty tree object, if Git were to be patched to support that representation.
Notwithstanding that, I do believe that the reason jj/xit/got use the same plumbing as Git is about compatibility.
Also, it’s not necessarily an issue with the newer generation. Young-ish programmers that I know are generally comfortable with Git, including me, while it seems like people used to Mercurial/SVN/CVS who have a hard time groking it.
As I said, I’m a git fanboy. I do think that the git object model is great, but I also see issues. One of my biggest issue is that git is incapable to track renames and copies.
If I rename a file and largely modify that file in the same commit, git will think that I deleted a file called
foo.txt, and committed a new different file (written from scratch) calledbar.txt. If I want to merge modifications offoo.txton the lines I didn’t touch, I will have merge conflicts.Copy is another issue. If I copy
foo.txttobar.txtand modifybar.txta little bit in one commit, the git object model will think that you just created a new filebar.txtfrom scratch. If you want to merge work that modifiedfoo.txtbefore you copied it, the changes will not be propagated tobar.txt.This is a very intentional decision by git, compared to other systems that tracked renames/copies with metadata (svn and hg, I think).
git is content-based, and the metadata is derived, rather than stored.
It is counterintuitive, but I think git’s choice is better, and I thought most VCS experts basically came around to that conclusion (?)
This rename and branch metadata issue was a big argument around 2007 when git was gaining popularity, but it seems to have been mostly settled …
I think one main problem is if you have a branch-heavy workflow, which turned out to be extremely useful in practice, then the most general solution is for renames to be derived from content. It just does a better job in practice.
Another under-rated thing is that git repositories are extremely backward compatible. A “branch” is almost nothing to the git model, just as a “rename” is literally nothing. This makes A LOT of things work more smoothly – the core is more elegant and compatible
I don’t personally run into that problem, and I rename all the time … I am not familiar with the details, but my guess is it’s because I commit very often
And also there are like a half a dozen merge strategies in git, and I guess the experts on big projects use those (but I don’t)
IME the alternatives always run into problems too … branches are cheap in git, but expensive in other systems – both in terms of time and cognitive overhead.
i.e. it doesn’t make sense to criticize the object model on theoretical grounds, when the systems that did the theoretically better thing actually behave worse in practice
I am not sure but Linus might have mentioned this at his ~2007 talk on git at Google, which was famously a bit abrasive (I was there):
https://www.youtube.com/watch?v=idLyobOhtO4
I think it’s one of those things where he had the contrary opinion at the time, was abrasive about it, and turned out to be right
Here is some rant I googled – I didn’t read the whole thing – but it is obviously extremely intentional and purposeful
https://gist.github.com/borekb/3a548596ffd27ad6d948854751756a08
And again, to change my mind, I would have to see a system that does better in practice … IME it’s a non-issue with git, and git does significantly better than its predecessors.
Most of the time using
git mvinstead ofmvworks for me although there are still a bunch of caveats. IMO this is a porcelain issue rather than an object model onePretty naturally, people who have used the feature of reliably-commit-attached branches have more issues with Git not allowing syncing the reflog than the people who were told no-real-branches is the only possibility and formed their model across this lack of functionality!
Yeah same here. To me, git just gets a lot of work done. It was a good 20%-50% permanent improvement over svn and hg in my mind, and that’s a huge win compounded over a decade.
I think many issues can be solved by simply committing more often, which I always did with every VCS. And the others issues I hit along ago and have workarounds for. Stack Overflow is invaluable here.
It’s funny to me that I don’t think I would be happy with git without Google/Stack Overflow, which is a sign of deficiency, but it’s still the best thing. (And yes I treat the command names and flags as opaque crap to memorize.)
I already spend too much time faffing with various tools, so for me git and vim are “set and forget”. On the other hand, I seem to have more problems with distros (Ubuntu was sort of ruined IMO, Debian is OK but can use improvement), and browsers (Firefox and Chromium both have issues).
Interesting, I used to think the same way. I considered myself a git power user and had no problems with it at all. I was very productive, comfortable with editing my history and could fix all my coworkers’ problems. Then jj clicked for me and I couldn’t go back.
I was still slightly stuck with git because it’s more performant in huge repos like the kernel. So I frantically searched for ways to get jj’s productivity and power in git. I finally found the config option
rebase.rebaseMerges, which is the crucial missing piece to mimic the megamerge workflow of jj. Other config options are obviously essential too, butrebaseMergesis never really talked about online as far as I can tell.It’s not at all perfect. git is still more manual work and more friction than jj. And some things still annoyingly don’t work at all. (Why can’t git do an octopus merge when there is a conflict?? jj can.)
So git can definitely be usable and productive. But if a power user like me, with a big config polished over years, cannot get the same power that jj gives you out-of-the boxes and with an intuitive CLI… I definitely see jj as the future “default” VCS people will be using and recommending.
While git is certainly cumbersome from time to time, I’ve learned to live with it. But it doesn’t stop to amaze me how many people just put up with GitHub. Pushing a branch, going to web, clicking a button, filling out a form? Why isn’t pushing a branch enough? And then reviewing online in a GUI so slow it takes a second to open a side-panel with the list of affected files, at least on large PRs. This all feels so backwards.
It’s possible to check out GitHub PRs locally…
Except that if you merge that ref (or the actual source ref) directly with Git, GitHub won’t detect the PR as being merged
What a headache
I have to agree, I have no problems with git.
All these alternative systems seems to come from people who want simpler tools and do not understand git.
Do you really think authors of these alternative systems do not understand git? People like @martinvonz?
The allocator design space is huge and nuanced, so at a cursory glance I’m taking the strong claims by the blog post and related PRs/issues with a little skepticism. Definitely looking forward to more rigorous comparisons with other widely used/SOTA allocators!
EDIT: Forgot to mention that the code is readable and easily understood even though I don’t write Zig, really recommend at least perusing the source!
Yeah, one obvious thing is that I don’t see any measure of fragmentation. If you use more space, you can save time.
https://ziglang.org/devlog/2025/#2025-02-07
I only see 2 workloads there – a synthetic benchmark, and a Zig compiler workload. Allocator performance is VERY multi-dimensional, so you need more workloads to draw any conclusions
e.g. test on say Firefox, Redis, and SPEC benchmarks, like this paper:
https://people.cs.umass.edu/~mcgregor/papers/19-pldi.pdf
and measure both space and time. Different programs have very different allocation patterns
Otherwise I’d just say “outcompetes glibc malloc on the things I tried”, not “outcompetes glibc malloc”.
I wouldn’t say that that Oils has an allocator that beats glibc too, even though here it runs 74.5% of the time as glibc (IOW more than 25% faster), as measured by cachegrind.
and on another workload it runs in 69.3% of the time of glibc:
https://oils.pub/release/0.27.0/benchmarks.wwz/gc-cachegrind/ (these measurements include GC time, but there is also a mode to exclude GC, and I know it’s faster in both modes)
Looking at the code, it’s 124 lines of C++.
Critical thinking exercise: why is this NOT surprising? (and arguably, not impressive)
Also, while doing that work, I observed that glibc free() is often slower than malloc().
Periodically, there would be glibc functions called from free() that showed up at the TOP of the perf profiles.
Which I now know to be “coalescing”, i.e. to reduce fragmentation
So you can make a faster allocator than glibc simply by removing the coalescing, at the cost of space.
The Zig allocator is not doing any coalescing
https://sourceware.org/glibc/wiki/MallocInternals - some details on glibc internals
Claude AI claims that glibc uses Knuth’s boundary tag coalescing, described in 1973 in TaoCP. The wiki page doesn’t say that (it seems plausible from what I read?), but that is a real thing:
https://www.cs.cmu.edu/afs/cs/academic/class/15213-s12/www/lectures/18-allocation-basic-1up.pdf
But this is actually separate from the point I was trying to make, which is that we “just” wrote a pool allocator in 124 lines of code (done by Chris Watkins)
And based on knowledge of the workload, we have a dirt simple policy - there are 2 pools, and one is 24 bytes, and one is 48 bytes. That’s a lot faster than glibc for our workload.
So it’s easy to beat glibc on particular workloads
mimalloc was also slower for us than this scheme - https://github.com/microsoft/mimalloc
I’ve heard the claim that “there is no such thing as general purpose memory allocation”, and I tend to believe that … malloc/free is a bare bones interface that can be used in a bazillion ways! There are many dimensions
All that said, I’m not arguing this isn’t a good allocator for certain programs (“idiomatic Zig” ?). And I’d bet that glibc was written / tuned in the days when hardware was much different, e.g. cache misses were 10x-100x less expensive, relatively speaking
Thanks for organizing this event! It was a good use of time – the talks were interesting and well paced
I’d personally be biased toward more “experience reports” than “new ideas”, but I think it’s a good format, and maybe others can take some cues for a variety of events in the same format
I wouldn’t have minded some more organized discussion afterward, although I don’t have any ideas on how to make that happen
I’m also glad Zulip worked out, since TBH if it was on Slack or Discord I may not have joined :)
Yeah, it’s a weird common knowledge problem. There could have been a big discussion the day afterwards if everyone believed there was going to be one. I have thought about spreading the talks out more to give the discussion more room to breathe. Maybe one block per evening or similar.
This has also been my understanding of X as Code for the better part of a decade - it wasn’t until this year that I realized there are people who understood this differently and disagreed with this definition as passionately as they do.
Interesting! I’ve never cared to wade into it, but I always read “config as code” to mean turing complete as opposed to declarative. Another score for using ambiguous terms.
What is the contrary view? Who are its proponents?
The contrary view is mostly a variation of “configuration is not programming”, which I guess defines “code” vaguely as something Turing complete?
For example, there was this article posted here not long ago which made the point that Terraform is configuration, not code: https://xeiaso.net/blog/2025/yoke-k8s/. Somewhat expectedly, this lead to some discussion here: https://lobste.rs/s/t0uh3q/yoke_is_really_cool
Ugh these are all the same statement
I’ll add a bit of substance for anyone not convinced. For context, I’m a programmer that knows things like assembly, C, OCaml, Rust and what have you. I can promise you I have done Terraform programming using its more advanced features.
Terraform/HCL has:
Anyways all this is besides the point, using code/programming in a way that excludes things that don’t look like your favorite language is unnecessary gatekeeping.
“Code” doesn’t necessarily mean “programming language.” I see code as a superset of programming and configuration languages (and other things like spreadsheets or small plastic bricks).
And the programming and config lang sets have some overlap because, as I said in the other thread, configuration is semantics and can be done in basically any language. For instance, Lua is not always configuration, but with Neovim it mostly is.
I think Mitchell’s aim was to get a description of the system
But there are things that can be programmed which cannot be programmed in this kind of code. Pegboard programming of synthesizers or the ENIAC, visual programming like Labview, mechanisms in Lego or Meccano.
(fun fact! the Computer Lab in Cambridge was founded in 1937 starting with equipment such as differential analysers built using meccano.)
There are other non-programming activities that can be turned into code, such as writing (are you using a word processor? or LaTeX?) or diagramming (are you using Inkscape? or Graphviz?).
Where “programming” and “code” is used interchangeably so they can gatekeep the profession.
Is this sentiment a consequence of people misusing “coding” and “coder” to mean programming and programmer? i.e. “all programming is coding” transforms into “all coding is programming” (not to say that declarative code isn’t programming). That word choice has always bothered me, it missed the point of what programming’s about.
How long before we can add
to your list?
I’m not sure what your point is. Surely you don’t think writing words down is inherently programming?
FWIW I divided config languages into 5 categories here, I view them as a spectrum from data to code
https://github.com/oils-for-unix/oils/wiki/Survey-of-Config-Languages
And I do think there are 2 separate ideas in “X as code”:
I’d also differentiate between code executed by courts and executed with determinism
What about code executed by multiple implementations of the interpreter?
This is not code. This is configuration.
FWIW we’ve been working on letting you declare data in YSH, a new Unix shell.
So you can arbitrarily interleave code and data, with the same syntax. The config dialect is called “Hay” - Hay Ain’t YAML.
Here’s a demo based on this example: https://github.com/oils-for-unix/blog-code/blob/main/hay/iac-demo.ysh
It looks almost the same as HCL (although I think this was convergent evolution, since I’ve actually never used Terraform):
And then the stdout of this config “program” is here - https://github.com/oils-for-unix/blog-code/blob/main/hay/output.txt
It can be serialized to JSON, or post-processed and then serialized
Then I show you can wrap Resource in a for loop, as well as parameterize it with a “proc” (procedure).
This is all still in progress, and can use feedback, e.g. on Github. (This demo runs, but it relies on a recent bug fix.)
The idea is not really to make something like Terraform, but rather to make a language with metaprogramming powerful enough to make your own “dialects”, like Terraform.
I wrote a doc about Hay almost 3 years ago - Hay - Custom Languages for Unix Systems - https://oils.pub/release/0.27.0/doc/hay.html
Comments - https://lobste.rs/s/phqsxk/hay_ain_t_yaml_custom_languages_for_unix
At that time, Oils was a slow Python prototype, but now it’s fast C++! So it’s getting there
The idea of Oils is shell+Python+JSON+YAML, squished together in the same language. So this works by reflection and function calls, not generating text (“Unix sludge”). No Go templates generating YAML, etc.