The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
(dev lead of TypeScript here, hi!)
We definitely knew when choosing Go that there were going to be people questioning why we didn’t choose Rust. It’s a good question because Rust is an excellent language, and barring other constraints, is a strong first choice when writing new native code.
Portability (i.e. the ability to make a new codebase that is algorithmically similar to the current one) was always a key constraint here as we thought about how to do this. We tried tons of approaches to get to a representation that would have made that port approach tractable in Rust, but all of them either had unacceptable trade-offs (perf, ergonomics, etc.) or devolved in to “write your own GC”-style strategies. Some of them came close, but often required dropping into lots of unsafe code, and there just didn’t seem to be many combinations of primitives in Rust that allow for an ergonomic port of JavaScript code (which is pretty unsurprising when phrased that way - most languages don’t prioritize making it easy to port from JavaScript/TypeScript!).
In the end we had two options - do a complete from-scrach rewrite in Rust, which could take years and yield an incompatible version of TypeScript that no one could actually use, or just do a port in Go and get something usable in a year or so and have something that’s extremely compatible in terms of semantics and extremely competitive in terms of performance.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions). We still want a highly-separated API surface to keep our implementation options open, so Go’s interop shortcomings aren’t particularly relevant. Go has excellent code generation and excellent data representation, just like Rust. Go has excellent concurrency primitives, just like Rust. Single-core performance is within the margin of error. And while there might be a few performance wins to be had by using unsafe code in Go, we have gotten excellent performance and memory usage without using any unsafe primitives.
In our opinion, Rust succeeds wildly at its design goals, but “is straightforward to port to Rust from this particular JavaScript codebase” is very rationally not one of its design goals. It’s not one of Go’s either, but in our case given the way we’ve written the code so far, it does turn out to be pretty good at it.
And it’s not even super clear what the upside of doing that would be (apart from not having to deal with so many “Why didn’t you choose Rust?” questions)
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Now that I’ve worked on a garbage collector, I see a sweet spot in languages like Go and C# – they have both value types deallocated on the stack and GC. Both Java and Python lack this semantic, so the GCs have to do more work, and the programmer has less control.
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a Foo* type as well as a Foo type, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.
I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between struct (value) and mutable struct (reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.
I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
$ go build -gcflags "-m" main.go
.\main.go:8:14: *y escapes to heap
.\main.go:11:13: x does not escape
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the node.parent reference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forward
Hejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with Object.freeze it isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have a node.parent reference.
So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the node.parent reference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.
The bad part is that then all the nodes you didn’t replace have chains of node.parent references that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutate node.parent, which means that 1) you can’t actually Object.freeze(node) and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.
See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the node.parent reference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.
Hm yeah it was a very good talk. My summary of the type checking part is
The input to the type checker is immutable ASTs
That is, parsing is “embarassingly parallel”, and done per file
They currently divide the program into 4 parts (e.g. 100 files turns into 4 groups of 25 files), and they do what I’d call “soft sharding”.
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess you have to de-duplicate the type errors? Because some type errors might come twice, since you are duplicating some work
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Why the sharding is in 4 parts, and not # CPUs. Even dev machines have 8-16 cores these days, and servers can have 64-128 cores.
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
Under 8 cores and 8 threads, the parallel front end can reduce the clean build (cargo build with -Z threads=8 option) time by about 30% on average. (These crates are from compiler-benchmarks of rustc-perf)
I guess this is just because, empirically, you don’t get more than 3x speedup.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
Acknowledging some weak spots, Go’s in-proc JS interop story is not as good as some of its alternatives. We have upcoming plans to mitigate this, and are committed to offering a performant and ergonomic JS API.
There was an issue with them that caused them to not work well in nix pkgs (need to find the issue) but they should be fine for things outside nix pkgs
The NixOS project also had to deal with this. Equinix Metal (formally Packet, RIP) was the primary infra provider for the various forms of CI we use (namely our build farm and PR autobuilds), and it’s taken us a bit of time to get things in place to switch to different providers, but we’re more or less done now.
I am slightly surprised fdo didn’t talk about this sooner — it’s been known that Metal is shutting down for three-ish months at this point, which would give them six months to plan this out instead of three. Hugs to the fdo sysops who have to do this with an accelerated timeline.
Happy to see this. I used Puppet extensively for many years, and it’s possibly the best of the config management tools out there. I do think this move is perhaps a bit too late, as even five years ago the writing was on the wall for the decline of their open source community. I have moved on to NixOS, but I wish the team well!
At a former job we used Puppet to manage tens of thousands of servers and it mostly worked fine. It was not even unpleasant to change or figure out, and after trying out Ansible and Terraform I came away convinced that the pull model is better for server management than the push model.
On my personal machines I use NixOS now also, and you will pry it from my cold dead hands.
I also use NixOS for my personal machines, and would love for a way to make the pull model work there too. Right now, all the popular deploy mechanisms for Nix are push-based (colmena, deploy-rs, –target-host…)
The “reply to opt out” relicensing scheme is not something I’ve seen before – I’ve only ever seen “reply to opt in” relicensing, where maintainers wait for all contributors to explicitly OK it.
I wonder if something like this would actually hold up from a legal standpoint.
I wonder if something like this would actually hold up from a legal standpoint.
As noted in the RFC discussion, this would by no means be “OK”-ed by any lawyer as there is ample opportunity to litigate if you really wanted too.
What we are doing is hinging a bet on most of this work not being copyrightable, along with contributors being reasonable about their contributions. If this is not the case people have ample opportunity to invoke their copyright and we would spend the required effort to fix the copyright infringement. We consider this a better balance on risk versus protecting maintainer time and effort.
I also doubt anyone would seriously litigate against package files, which has been remixed, rewritten, translated, borrowed and utilized across multiple distros over 20 decades.
Another example which does what you mentioned, was when we realized our developer tooling had no license and we had to fetch approval from all past contributors.
I’m now jobless as last week (“yay”), so trying to figure things out. I saw the writing on the wall for a while, and luckily got ahead of sending in some applications, so hopefully I’ll have better news in the next few weeks.
Will be nice to have more time to review PRs and such, at least.
Not needing a C compiler on your build machine + not having the (infrequent, to be fair) hideously long build times you get when using mattn/go-sqlite3.
Thanks, I think you’re right and that cgo is the main reason to skip mattn/go-sqlite3 in some contexts. :)
On the other hand, modernc.org/sqlite is advertised as a “cgo-free” sqlite driver, so if “no-cgo” was the goal, maybe GtS could have used this instead of mattn’s driver. I would understand the need for WASM if it improved performance over the others, but this article seems to contradict that idea a bit (at least regarding memory, these numbers look very weird).
So I’m still confused, sorry. Maybe I need more caffeine. :)
Frustrating to read to the end of this “introduction” without any idea of what this text format looks like. In the end, it’s just a page on a web site built by a tool we can’t see yet? This feels premature to be posting it.
Side note: you may want to rethink “dtf” as your file extension.
I found a “view source” link at the bottom of the page. So this is what it looks like. Wish the article about their markup language had side-by-side comparisons, though; I kept tabbing back and forth between the article and source pages.
Flagged off-topic; this is internal community management stuff, which is a bit off-topic I believe. I also suspect it’ll cause discussion here that generates more heat than light.
I haven’t followed the various nix stories much beyond skimming what’s been posted here, but this is anodyne process stuff that follows pretty naturally from those. I don’t take it as trying to brigade the project. I don’t like the heckler’s veto and I’m definitely not going to preemptively give in to it.
For what it’s worth, my main reason for flagging is that the voting itself is limited to only a portion of the readership here, though I suppose the election process and automation itself may be of interest to onlookers; perhaps I was wrong to flag.
For a while I used proxmox but I ended up always having trouble every time it was upgraded. So I simply moved over to Alpine with libvirt and qemu and I’ve never looked back. Upgrades are very simple. Just change a simple line in my file and APK upgrade minus minus available. It works really well. I spend very little time on it and it just runs and automatically patches itself. I will admit that I really prefer wholesale internet for most of my remote stuff. They’re cheap and have excellent service. If you are not us-based you might not like them because they are in Kansas. I’ve been thinking about using incus instead, but that’s a project for a different year. I’d rather be out biking on trike recumbent.
Oh, they look potentially neat– ahahaha wow I used to work in a datacenter literally a few blocks away from them. Kansas City is one of the big east-west network arteries in the US, if what I’ve been told is true, so it’s a good place for networking infrastructure.
For folks already into pkgsrc (please note hat and sprinkle grains of salt accordingly), it’s pretty easy to bootstrap multiple prefixes and give each its own Python. For a solo project I might just do that. For team projects, I’d probably want us to converge on asdf or mise wherever possible (and I maintain pkgsrc’s packages of both).
$ ./bootstrap --unprivileged ~/pkg/ancient-project
$ oldpath="$PATH"
$ PATH="~/pkg/ancient-project/bin:~/pkg/ancient-project/sbin:$PATH"
$ cd ../lang/python27 && bmake install
$ # also install anything else needed for ancient-project
$ PATH="$oldpath"
$ ./bootstrap --unprivileged ~/pkg/recent-project
$ oldpath="$PATH"
$ PATH="~/pkg/recent-project/bin:~/pkg/recent-project/sbin:$PATH"
$ cd ../lang/python312 && bmake install
$ # also install anything else needed for recent-project
$ PATH="$oldpath"
Then when you’re working on one of these projects, set PATH accordingly, maybe with direnv. (Assuming a non-project-specific general-purpose pkgsrc installation already in $PATH, install direnv in it.)
I definitely dislike the format Lennart has chosen for announcing these changes. Due to Mastodon’s character limit, each installment is spread over several posts. Reading the whole set of posts is truly a hassle.
The most fascinating thing to me is that for something apparently launched in 2024 they still unambiguously use @… to only mean x/twitter and not link the account, while claiming “for developers”.
Not that I want to derail this into another Twitter vs Fediverse thread, but I’m still very surprised as my bubble has completely moved on.
Archive.org shows it linking to Twitter in all recent grabs; maybe the lack of underline/hover effects caused confusion.
edit: actually, wink might’ve been referring to the links on the faq page. The “readme” (/about) page’s @mentions do, though. It all feels very … something.
I’d love to see the source just for a good example of proper use of PostgreSQL for full-text indexing! I haven’t seen an application that does full-text indexing/search so well, and so quickly, in a very long time. I have a feeling that, even if it’s got some of the usual small hacks that personal projects sometimes have, this is the kind of source code that aspiring developers would find extremely rewarding to read and understand.
Thank you! Unfortunately not, it’s more of a 48 hour hack that is too ugly and way too tailored for the OpenBSD manual pages. I hope to be able to release it in a somewhat near future though, when most of the ugly stuff is sorted.
The TypeScript dev lead posted this response about the language choice on Reddit, for anyone who’s curious:
Source: https://www.reddit.com/r/typescript/comments/1j8s467/comment/mh7ni9g
People really miss the forest for the trees.
I looked at the repo and the story seems clear to me: 12 people rewrote the TypeScript compiler in 5 months, getting a 10x speed improvement, with immediate portability to many different platforms, while not having written much Go before in their lives (although they are excellent programmers).
This is precisely the reason why Go was invented in the first place. “Why not Rust?” should not be the first thing that comes to mind.
I honestly do think the “Why not Rust?” question is a valid question to pop into someone’s head before reading the explanation for their choice.
First of all, if you’re the kind of nerd who happens to follow the JavaScript/TypeScript dev ecosystem, you will have seen a fair number of projects either written, or rewritten, in Rust recently. Granted, some tools are also being written/rewritten in other languages like Go and Zig. But, the point is that there’s enough mindshare around Rust in the JS/TS world that it’s fair to be curious why they didn’t choose Rust while other projects did. I don’t think we should assume the question is always antagonistic or from the “Rust Evangelism Strike Force”.
Also, it’s a popular opinion that languages with algebraic data types (among other things) are good candidates for parsers and compilers, so languages like OCaml and Rust might naturally rank highly in languages for consideration.
So, I honestly had the same question, initially. However, upon reading Anders’ explanation, I can absolutely see why Go was a good choice. And your analysis of the development metrics is also very relevant and solid support for their choice!
I guess I’m just saying, the Rust fanboys (myself, included) can be obnoxious, but I hope we don’t swing the pendulum too far the other way and assume that it’s never appropriate to bring Rust into a dev conversation (e.g., there really may be projects that should be rewritten in Rust, even if people might start cringing whenever they hear that now).
While tweaking a parser / interpreter a few years ago written in Go, I specifically replaced a struct with an ‘interface {}’ in order to exercise its pseudo-tagged-union mechanisms. Together with using type-switch form.
https://github.com/danos/yang/commit/c98b220f6a1da7eaffbefe464fd9e734da553af0
These day’s I’d actually make it a closed interface such that it is more akin to a tagged-union. Which I did for another project which was passing around instances of variant-structs (i.e. a tagged union), rather than building an AST.
So it is quite possible to use that pattern in Go as a form of sum-type, if for some reason one is inclined to use Go as the implementation language.
That is great explanation of “Why Go and not Rust?”
If you’re looking for “Why Go and not AOT-compiled C#?” see here: https://youtu.be/10qowKUW82U?t=1154s
A relevant quote is that C# has “some ahead-of-time compilation options available, but they’re not on all platforms and don’t really have a decade or more of hardening.”
That interview is really interesting, worth watching the whole thing.
Yeah Hjelsberg also talks about value types being necessary, or at least useful, in making language implementations fast
If you want value types and automatically managed memory, I think your only choices are Go, D, Swift, and C# (and very recently OCaml, though I’m not sure if that is fully done).
I guess Hjelsberg is conceding that value types are a bit “second class” in C#? I think I was surprised by the “class” and “struct” split, which seemed limiting, but I’ve never used it. [1]
And that is a lesson learned from the Oils Python -> C++ translation. We don’t have value types, because statically typed Python doesn’t, and that puts a cap on speed. (But we’re faster than bash in many cases, though slower in some too)
Related comment about GC and systems languages (e.g. once you have a million lines of C++, you probably want GC): https://lobste.rs/s/gpb0qh/garbage_collection_for_systems#c_rrypks
There was also a talk that hinted at some GC-like patterns in Zig, and I proposed that TinyGo get “compressed pointers” like Hotspot and v8, and then you would basically have that:
https://lobste.rs/s/2ah6bi/programming_without_pointers#c_5g2nat
[1] BTW Guy Steele’s famous 1998 “growing a language” actually advocated value types in Java. AFAIK as of 2025, “Project Valhalla” has not landed yet
Compilers written in OCaml are famous for being super-fast. See eg OCaml itself, Flow, Haxe, BuckleScript (now ReScript).
Yeah, I’m kind of curious about whether OCaml was considered at some point (I asked about this in the Reddit thread, haven’t gotten a reply yet).
OCaml seems much more similar to TS than Go, and has a proven track record when it comes to compilers. Maybe portability issues? (Good portability was mentioned as a must-have IIRC)
Maybe, but given that Flow, its main competitor, distributes binaries for all major platforms: https://github.com/facebook/flow/releases/tag/v0.264.0
Not sure what more TypeScript would have needed. In fact, Flow’s JavaScript parser is available as a separate library, so they would have shaved off at least a month from the proof of concept…
Also Nim.
Also Julia.
There surely are others.
Yes good points, I left out Nim and Julia. And apparently Crystal - https://colinsblog.net/2023-03-09-values-and-references/
Although thinking about it a bit more, I think Nim, Julia, (and maybe Crystal) are like C#, in that they are not as general as Go / D / Swift.
You don’t have a
Foo*type as well as aFootype, i.e. the layout is orthogonal to whether it’s a value or reference. Instead, Nim apparently has value objects and reference objects. I believe C# has “structs” for values and classes for references.I think Hjelsberg was hinting at this category when saying Go wins a bit on expressiveness, and it’s also “as close to native as you can get with GC”.
I think the reason this Go’s model is uncommon is because it forces the GC to support interior pointers, which is a significant complication (e.g. it is not supported by WASM GC). Go basically has the C memory model, with garbage collection.
I think C#, Julia, and maybe Nim/Crystal do not support interior pointers (interested in corrections)
Someone should write a survey of how GC tracing works with each language :) (Nim’s default is reference counting without cycle collection.)
Yeah that’s interesting. Julia has a distinction between
struct(value) andmutable struct(reference). You can use raw pointers but safe interior references (to an element of an array for example) include a normal reference to the (start of the) backing array, and the index.I can understand how in Rust you can safely have an interior pointer as the borrow checker ensures a reference to an array element is valid for its lifetime (the array can’t be dropped or resized before the reference is dropped). I’m very curious - I would like to understand how Go’s tracing GC works with interior pointers now! (I would read such a survey).
Ok - Go’s GC seems to track a memory span for each object (struct or array), stored in kind of a span tree (interval tree) for easy lookup given some pointer to chase. Makes sense. I wonder if it smart enough to deallocate anything dangling from non-referenced elements of an array / fields of a struct, or just chooses to be conservative (and if so do users end up accidentally creating memory leaks very often)? What’s the performance impact of all of this compared to runtimes requiring non-interior references? The interior pointers themselves will be a performance win, at the expense of using an interval tree during the mark phase.
https://forum.golangbridge.org/t/how-gc-handles-interior-pointer/36195/5
It’s been a few years since I’ve written any Go, but I have a vague recollection that the difference between something being heap or stack allocated was (sometimes? always?) implicit based on compiler analysis of how you use the value. Is that right? How easy it, generally, to accidentally make something heap-allocated and GC’d?
That’s the only thing that makes me nervous about that as a selling point for performance. I feel like if I’m worried about stack vs heap or scoped vs memory-managed or whatever, I’d probably prefer something like Swift, Rust, or C# (I’m not familiar at all with how D’s optional GC stuff works).
Yes, that is a bit of control you give up with Go. Searching for “golang escape analysis”, this article is helpful:
https://medium.com/@trinad536/escape-analysis-in-golang-fc81b78f3550
So the toolchain is pretty transparent. This is actually something I would like for the Oils Python->C++ compiler, since we have many things that are “obviously” locals that end up being heap allocated. And some not so obvious cases. But I think having some simple escape analysis would be great.
Yes, the stack/heap distinction is made by the compiler, not the programmer, in Go.
Why did you leave JS/TS off the list? They seem to have left it off too and that confuses me deeply because it also has everything they need
Hejlsberg said they got about 3x performance from native compilation and value types, which also halved the memory usage of the compiler. They got a further 3x from shared-memory multithreading. He talked a lot about how neither of those are possible with the JavaScript runtime, which is why it wasn’t possible to make tsc 10x faster while keeping it written in TypeScript.
Yeah but I can get bigger memory wins while staying inside JS by sharing the data structures between many tools that currently hold copies of the same data: the linter, the pretty-formatter, the syntax highlighter, and the type checker
I can do this because I make my syntax tree nodes immutable! TS cannot make their syntax tree nodes immutable (even in JS where it’s possible) because they rely on the
node.parentreference. Because their nodes are mutable-but-typed-as-immutable, these nodes can never safely be passed as arguments outside the bounds of the TS ecosystem, a limitation that precludes the kind of cross-tool syntax tree reuse that I see as being the way forwardHejlsberg said that the TypeScript syntax tree nodes are, in fact, immutable. This was crucial for parallelizing tsgo: it parses all the source files in parallel in the first phase, then typechecks in parallel in the second phase. The parse trees from the first phase are shared by all threads in the second phase. The two phases spread the work across threads differently. He talks about that kind of sharing and threading being impractical in JavaScript.
In fact he talks about tsc being designed around immutable and incrementally updatable data structures right from the start. It was one of the early non-batch compilers, hot on the heels of Roslyn, both being designed to support IDEs.
Really, you should watch the interview https://youtu.be/10qowKUW82U
AIUI a typical LSP implementation integrates all the tools you listed so they are sharing a syntax tree already.
It’s true that I haven’t watched the interview yet, but I have confirmed with the team that the nodes are not immutable. My context is different than Hejlsberg’s context. For Hejlsberg if something is immutable within the boundaries of TS, it’s immutable. Since I work on JS APIs if something isn’t actually locked down with
Object.freezeit isn’t immutable and can’t safely be treated as such. They can’t actually lock their objects down because they don’t actually completely follow the rules of immutability, and the biggest thing they do that you just can’t do with (real, proper) immutable structures is have anode.parentreference.So they have this kinda-immutable tech, but those guarantees only hold if all the code that ever holds a reference to the node is TS code. That is why all this other infrastructure that could stand to benefit from a shared standard format for frozen nodes can’t: it’s outside the walls of the TS fiefdom, so the nodes are meant to be used as immutable but any JS code (or any-typed code) the trees are ever exposed to would have the potential to ruin them by mutating the supposedly-immutable data
To be more specific about the
node.parentreference, if your tree is really truly immutable you need to replace a leaf node you must replace all the nodes on the direct path from the root to that leaf. TS does this, which is good.The bad part is that then all the nodes you didn’t replace have chains of
node.parentreferences that lead to the old root instead of the new one. Fixing this with immutable nodes would mean replacing every node in the tree, so the only alternative is to mutatenode.parent, which means that 1) you can’t actuallyObject.freeze(node)and 2) you don’t get all the wins of immutability since the old data structure is corrupted by the creation of the new one.See https://ericlippert.com/2012/06/08/red-green-trees/ for why Roslyn’s key innovation in incremental syntax trees was actually breaking the
node.parentreference by splitting into the red and green trees, or as I call them paths and nodes. Nodes are deeply immutable trees and have no parents. Paths are like an address in a particular tree, tracking a node and its parents.You are not joking, just the hack to make type checking itself parallel is well worth an entire hour!
Hm yeah it was a very good talk. My summary of the type checking part is
That is, the translation units aren’t completely independent. Type checking isn’t embarassingly parallel. But you can still parallelize it and still get enough speedup – he says ~3x from parallelism, and ~3x from Go’s better single core perf, which gives you ~10x overall.
What wasn’t said:
I guess this is just because, empirically, you don’t get more than 3x speedup.
That is interesting, but now I think it shows that TypeScript is not designed for parallel type checking. I’m not sure if other compilers do better though, like Rust (?) Apparently rustc uses the Rayon threading library. Though it’s hard to compare, since it also has to generate code
A separate thing I found kinda disappointing from the talk is that TypeScript is literally what the JavaScript code was. There was never a spec and will never be one. They have to do a line-for-line port.
There was somebody who made a lot of noise on the Github issue tracker about this, and it was basically closed “Won’t Fix” because “nobody who understands TypeScript well enough has enough time to work on a spec”. (Don’t have a link right now, but I saw it a few months ago)
Pretty sure he said it was an arbitrary choice and they’d explore changing it. The ~10x optimization they’ve gotten so far is enough by itself to keep the project moving. Further optimization is bound to happen later.
Work has been going on for years to parallelize rust’s frontend, but it apparently still has some issues, and so isn’t quite ready for prime time just yet, though it’s expected to be ready in the near term.
In my experience, once you start to do things “per core” and want to actually get performance out of it, you end up having to pay attention to caches, and get a bit into the weeds. Given just arbitrarily splitting up the work as part of the port has given a 10x speed increase, it’s likely they just didn’t feel like putting in the effort.
Can you share the timestamp to the discussion of this hack, for those who don’t have one hour?
I think this one: https://www.youtube.com/watch?v=10qowKUW82U&t=2522s
But check the chapters, they’re really split into good details. The video is interesting anyway, technically focused, no marketing spam. I can also highly recommend watching it.
Another point on “why Go and not C#” is that, he said, their current (typescript) compiler is highly functional, they use no classes at all. And Go is “just functions and data structures”, where C# has “a lot of classes”. Paraphrasing a little, but that’s roughly what he said.
They also posted a (slightly?) different response on GitHub: https://github.com/microsoft/typescript-go/discussions/411
Yes please!
URL should be updated to https://jfmengels.net/multi-file-fixes.
I think file sets would help with this case https://nixos.org/manual/nixpkgs/stable/#sec-functions-library-fileset
There was an issue with them that caused them to not work well in nix pkgs (need to find the issue) but they should be fine for things outside nix pkgs
See https://github.com/NixOS/nix/issues/11503 for the actual Nix issue causing them to be banned in Nixpkgs, and https://github.com/NixOS/nixpkgs/pull/369694 for further context.
The NixOS project also had to deal with this. Equinix Metal (formally Packet, RIP) was the primary infra provider for the various forms of CI we use (namely our build farm and PR autobuilds), and it’s taken us a bit of time to get things in place to switch to different providers, but we’re more or less done now.
I am slightly surprised fdo didn’t talk about this sooner — it’s been known that Metal is shutting down for three-ish months at this point, which would give them six months to plan this out instead of three. Hugs to the fdo sysops who have to do this with an accelerated timeline.
Happy to see this. I used Puppet extensively for many years, and it’s possibly the best of the config management tools out there. I do think this move is perhaps a bit too late, as even five years ago the writing was on the wall for the decline of their open source community. I have moved on to NixOS, but I wish the team well!
At a former job we used Puppet to manage tens of thousands of servers and it mostly worked fine. It was not even unpleasant to change or figure out, and after trying out Ansible and Terraform I came away convinced that the pull model is better for server management than the push model.
On my personal machines I use NixOS now also, and you will pry it from my cold dead hands.
I also use NixOS for my personal machines, and would love for a way to make the pull model work there too. Right now, all the popular deploy mechanisms for Nix are push-based (colmena, deploy-rs, –target-host…)
Perhaps you’ll like https://github.com/nlewo/comin or https://github.com/rapenne-s/bento, two pull-based deploy solutions for NixOS :)
Oh very cool ! Going to check them out. Thank you
This is why my main side project is experimenting with pull-based NixOS deployment. ;)
Literally clearing my mental space to propose to my future wife; let’s hope she says yes.
Other than that, finishing a few books I had ([1], [2], [3], [4]) in my queue and trying to focus on what matters: myself and others around me
[1] Genius Makers [2] The Geek Leadership Handbook [3] The Structure of the Mind [4] Why Greatness Cannot Be Planned
Congratulations!
Good luck :)
Wish you the best of luck and courage man!
Good luck!
The “reply to opt out” relicensing scheme is not something I’ve seen before – I’ve only ever seen “reply to opt in” relicensing, where maintainers wait for all contributors to explicitly OK it.
I wonder if something like this would actually hold up from a legal standpoint.
As noted in the RFC discussion, this would by no means be “OK”-ed by any lawyer as there is ample opportunity to litigate if you really wanted too.
What we are doing is hinging a bet on most of this work not being copyrightable, along with contributors being reasonable about their contributions. If this is not the case people have ample opportunity to invoke their copyright and we would spend the required effort to fix the copyright infringement. We consider this a better balance on risk versus protecting maintainer time and effort.
I also doubt anyone would seriously litigate against package files, which has been remixed, rewritten, translated, borrowed and utilized across multiple distros over 20 decades.
Another example which does what you mentioned, was when we realized our developer tooling had no license and we had to fetch approval from all past contributors.
https://github.com/archlinux/devtools/pull/62
I’m now jobless as last week (“yay”), so trying to figure things out. I saw the writing on the wall for a while, and luckily got ahead of sending in some applications, so hopefully I’ll have better news in the next few weeks.
Will be nice to have more time to review PRs and such, at least.
I might try to hack on Bluesky support for Lobsters.
Voluntary or no? Either way I sympathize. In that boat right now. Good luck out there o7
Involuntary I’m afraid :(
Good luck to you too! Hopefully we’ll both have better news soon! :)
Good luck with your applications!
I’m sorry if that is off-topic, but what does GtS gain from having a WASM-based sqlite driver?
Not needing a C compiler on your build machine + not having the (infrequent, to be fair) hideously long build times you get when using
mattn/go-sqlite3.Thanks, I think you’re right and that cgo is the main reason to skip
mattn/go-sqlite3in some contexts. :)On the other hand,
modernc.org/sqliteis advertised as a “cgo-free” sqlite driver, so if “no-cgo” was the goal, maybe GtS could have used this instead of mattn’s driver. I would understand the need for WASM if it improved performance over the others, but this article seems to contradict that idea a bit (at least regarding memory, these numbers look very weird).So I’m still confused, sorry. Maybe I need more caffeine. :)
My understanding was that GtS does use modernc’s sqlite, and this is why it’s broken on OpenBSD (for now?).
Checked and it looks like there’s a build-time choice between
modernc.org/sqlite(transcoded Go) andgithub.com/ncruces/go-sqlite3(wrapped WASM)Frustrating to read to the end of this “introduction” without any idea of what this text format looks like. In the end, it’s just a page on a web site built by a tool we can’t see yet? This feels premature to be posting it.
Side note: you may want to rethink “dtf” as your file extension.
The dtf file extension seems intentional, judging by the second email listed there.
I found a “view source” link at the bottom of the page. So this is what it looks like. Wish the article about their markup language had side-by-side comparisons, though; I kept tabbing back and forth between the article and source pages.
The Lobsters poster likely is not the author, FWIW.
Flagged off-topic; this is internal community management stuff, which is a bit off-topic I believe. I also suspect it’ll cause discussion here that generates more heat than light.
I haven’t followed the various nix stories much beyond skimming what’s been posted here, but this is anodyne process stuff that follows pretty naturally from those. I don’t take it as trying to brigade the project. I don’t like the heckler’s veto and I’m definitely not going to preemptively give in to it.
For what it’s worth, my main reason for flagging is that the voting itself is limited to only a portion of the readership here, though I suppose the election process and automation itself may be of interest to onlookers; perhaps I was wrong to flag.
How is this Nix related?
Apologies, couldn’t find anything related to package mangers, and the nix tag had
package manager, distribution, and related systems like guix.This is my first submission, So I should have been more careful. I have edited and removed the tag, Thanks!
Indeed; I think that’s meant to mean “[Nix] package manager, [NixOS] distribution, and related systems like guix.” Not necessarily super clear.
Off-topic(-ish), but the various pages on the author’s website are delightfully styled. It’s so nice to see.
For a while I used proxmox but I ended up always having trouble every time it was upgraded. So I simply moved over to Alpine with libvirt and qemu and I’ve never looked back. Upgrades are very simple. Just change a simple line in my file and APK upgrade minus minus available. It works really well. I spend very little time on it and it just runs and automatically patches itself. I will admit that I really prefer wholesale internet for most of my remote stuff. They’re cheap and have excellent service. If you are not us-based you might not like them because they are in Kansas. I’ve been thinking about using incus instead, but that’s a project for a different year. I’d rather be out biking on trike recumbent.
Oh, they look potentially neat– ahahaha wow I used to work in a datacenter literally a few blocks away from them. Kansas City is one of the big east-west network arteries in the US, if what I’ve been told is true, so it’s a good place for networking infrastructure.
I’m surprised I’ve never heard of WholeSaleInternet until now – will definitely keep in mind the next time I need a remote server, thank you!
Editing the 6 parts of my Kubernetes installation notes into a giant article for you all to enjoy. I suffered so you can read!
Very excited to read this, as someone who has never touched Kubernetes in their life.
For folks already into pkgsrc (please note hat and sprinkle grains of salt accordingly), it’s pretty easy to bootstrap multiple prefixes and give each its own Python. For a solo project I might just do that. For team projects, I’d probably want us to converge on
asdformisewherever possible (and I maintain pkgsrc’s packages of both).Out of curiosity, what does that look like?
Something like so:
Then when you’re working on one of these projects, set
PATHaccordingly, maybe withdirenv. (Assuming a non-project-specific general-purpose pkgsrc installation already in$PATH, installdirenvin it.)I definitely dislike the format Lennart has chosen for announcing these changes. Due to Mastodon’s character limit, each installment is spread over several posts. Reading the whole set of posts is truly a hassle.
It would be better if he was using an instance with a higher character limit – I’m pretty sure most (excluding mastodon.social) set a much higher one.
Shame that it seemingly hasn’t been developed for 8 months, it looks really neat.
The most fascinating thing to me is that for something apparently launched in 2024 they still unambiguously use @… to only mean x/twitter and not link the account, while claiming “for developers”.
Not that I want to derail this into another Twitter vs Fediverse thread, but I’m still very surprised as my bubble has completely moved on.
But that’s just it: it’s obviously just your bubble, right? To contextualize things, Mastodon is 100x smaller than Threads
It links to Twitter for me.
Archive.org shows it linking to Twitter in all recent grabs; maybe the lack of underline/hover effects caused confusion.
edit: actually, wink might’ve been referring to the links on the faq page. The “readme” (/about) page’s @mentions do, though. It all feels very … something.
Yeah I didn’t pay attention where - but I noticed it. I think the people had their accounts linked, but not the org/company.
Looks nice and clean. Is it free software? Can I use it for indexing and searching my own documentation and host it myself?
I’d love to see the source just for a good example of proper use of PostgreSQL for full-text indexing! I haven’t seen an application that does full-text indexing/search so well, and so quickly, in a very long time. I have a feeling that, even if it’s got some of the usual small hacks that personal projects sometimes have, this is the kind of source code that aspiring developers would find extremely rewarding to read and understand.
Thank you! Unfortunately not, it’s more of a 48 hour hack that is too ugly and way too tailored for the OpenBSD manual pages. I hope to be able to release it in a somewhat near future though, when most of the ugly stuff is sorted.
Release the ugly software!
I don’t think it’s ever too early to release the code - put a big warning notice on the README that it’s all a big hack and ship it anyway.
The value for me would be seeing what hacky tricks you used to get it to work. I couldn’t care less if it’s not polished yet.
Agreed, and some of the fun lies in letting other people find cool ways to remove the hacky stuff (if you want to accept contributions).
I had done a hack as well for manpages with sqlite https://lobste.rs/s/t7tuej/epilys_buke_toy_manpage_full_text_search
Feel free to play with it, since it’s free software.