As a D fan, I agree with many of the criticisms, however, several I want to respond to:
It taking 5 minutes in release mode to compile is largely down to LLVM, not the D frontend. Every compiler that uses it is this slow.
The heavy RAM use is because D always disables the GC during compilation, on the premise that you’d rather get results as fast as possible than save memory for a short-lived process. Turning the GC off and on with flags or user code has no effect on this, but you can turn it on with the -lowmem compiler flag. This will make the frontend slower, but if you’re already burning 80% of your runtime in LLVM optimization routines, it probably has no perceptible effect on build speed.
I don’t know what you mean with static foreach, are you trying to chain if statements with static foreach? That fundamentally cannot work in any compiler that treats else as syntactically part of the if compound statement, which is basically every C-like language. Just put a return/continue/break at the end of each if and chain regular if statements.
I’m not aware of any issues with mixins that would necessitate token strings. Link a bug report?
That said, you’re totally right about the codegen and debug info bugs, and about D not simply being “better C++”. (Also, named parameters should be coming any year now!) However, my prediction is that if you move to a smaller language, you will only encounter a greater volume of issues.
edit: I’m working on my own language btw ( https://neat-lang.github.io ), and I can confirm two things: unrestricted macros are pretty great, and so are named parameters. It’s very hard to go back to a language without them.
It taking 5 minutes in release mode to compile is largely down to LLVM, not the D frontend. Every compiler that uses it is this slow.
Rust people love to blame LLVM too, but it’s kind of a moot point. First, the user doesn’t care whether it’s slow because of LLVM or D. Second, LLVM is doing at least half of the work involved in compiling, if not more, so it’s unreasonable to expect it to take less time than the frontend. Third, Clang also uses LLVM, and compile times are nowhere near as bad. At Mozilla when they started adding Rust code into Firefox, that small fraction of the code took up a much larger portion of the total compile time–the vast majority of the code is still C++.
First, the user doesn’t care whether it’s slow because of LLVM or D.
Well, if you can switch to the gcc-based gdc or the independent-backend dmd and see an improvement, it is interesting to note. Though there’s a few challenges with those (gdc has no binary build for Windows so you have to build it yourself right now, and dmd has some bugs and poor optimizations compared to the others) it still shows you might have other options.
Clang is literally the frontend whose specific usage patterns LLVM is optimized around though. “Clang is fast” doesn’t equate to “LLVM is fast”, only “LLVM is fast(ish) if you use it precisely like Clang uses it.” (Which is, of course, not documented.)
Anyway I agree that this is a problem for ldc, but I still feel it’s worth emphasizing to clarify that it’s not a language issue.
It is a language issue if your frontend uses LLVM, but uses it sub-optimally. I agree 100% that it isn’t fair, but the user doesn’t care. They’ll just switch to a different language.
Very interesting read, thanks. I tried to find more specific information about Jai, but only found https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md. Is there a language specification somewhere? It’s particularly interesting how a compile time decrease of 10-100x is possible if the developer at the same time is “able to do anything during compilation”.
The primer is quite obsolete at this point and the language creator doesn’t have any public documentation as far as I know. The only places where you can see fresh information is on Jon’s twitch stream and probably on the streams of some of the other devs or people having access to the beta.
Thanks. I wonder how then the author of the posted article is able to migrate a 40kSLOC project to Jai. How would he know about the language details and where would he get the compiler and debugger from?
Jonathan Blow started development on this language in 2014 and has kept the compiler behind closed doors until December 2019, where a closed beta began. This closed beta is still ongoing and about two months ago I was invited to it.
Most of the people that were interested in the language I think followed Jon’s work for a while so they could have been exposed to how it works. He is working at the same time on the compiler and a game plus dev tools developed in the language, which he streams on and off.
However for someone completely new, the beta comes with extensive examples extensively commented by the author, so most of the features are detailed there.
It’s particularly interesting how a compile time decrease of 10-100x is possible if the developer at the same time is “able to do anything during compilation”.
I don’t know much about Jai, but I can tell you this is a real issue with D. D compiles quite quickly if you keep to basic code - I have some old (smaller) games that compile in a fraction of a second (~30ms), and even my larger things like a 50k line web server with integrated gui tend to compile in ~2.5 seconds. Not bad at all…
At the same time, I can write a 100 line program that will take a minute to compile. Or more. Why? Because it does a bunch of compile time execution. So when you start running code at compile time, the number of lines of the project stops mattering. The number of lines stops mattering when you can run loops! And moreover, the way you write the compile time function can radically change it. D’s CTFE engine, for example, is very slow when doing large string appends in a loop. Similar code, rewritten to pre-allocate the string and copy chunks into it, which takes more lines of code and loops through the data twice (once to determine length, second to copy it over), can run 10x faster! (Such tricks sometimes help normal code too, but it isn’t as dramatic as here due to the ctfe engine’s own quirks of implementation)
So I’m very skeptical of any claim of language compile speed performance with D, and I expect Jai is the same, since it so heavily depends on how you do the compile time things. (to be fair to the critics, one of the selling points is that you can do this cool compile time function call stuff, so then people get excited and use it heavily… then it kills the build speeds. but if you advertise both it is fair for people to expect you can have both. and you can… but only to an extent and only if you take care to implement it well. You might still end up with a pre-build step that caches the result file and you can do that with C++ too.)
So when you start running code at compile time, the number of lines of the project stops mattering.
I assume this code is run by a kind of interpreter integrated with the compiler; if so the same performance issues apply as with every interpreter; there are ways to write faster interpreters, up to using JIT compilation; I didn’t have a look at the specific D compiler, so I don’t know what it actually does with compile time execution.
I didn’t have a look at the specific D compiler, so I don’t know what it actually does with compile time execution
It is an AST interpreter that copies nodes any time a value is mutated. It is about the worst possible thing you can imagine that still manages to somehow work (I suspect this is why the OP saw gigabytes of ram uses as well as the long builds). You can definitely compare different implementations of those, and D’s can be significantly better in theory, but the point stands that comparing compile time per line of code is not a useful metric once you introduce any time of compile time function calls that can occur on one of those lines.
In addition to this, I think the most important reason we have so many vulnerabilities (and bugs in general) is completely disregarded in the hunt for “safe” code: culturally tolerated and even encouraged complexity.
Amen. I’m ending my current gig because of this, actually. Too few programmers have a taste for simplicity, let alone the will to achieve it. I personally find it alarming that I am consistently the most powerful simplifier in the room, just because I happen to hold the same Philosophy of Software Design as John Ousterhout. (Great book by the way.)
Dijkstra warned us long ago, complexity is supposed to be the bane of our existence. Yet most people seem either blind or resigned. And so we keep piling cruft on top of cruft. If I ever quit this job this will be the main cause.
I think at the current time anyone that can be bothered to mail Jon with a small blurb about wanting to get into the beta can get in. At one point people had invitations available to give others.
I kind of have to agree with WilhelmVonWeiner on this. That’s quite a barrier to entry compared to other languages, especially for one with a bunch of kruft from C and C++.
Are there any public projects written in Jai demonstrating why I’d jump through hoops to use it? The list in the blog reads like they copy/pasted marketing fluff, and the majority of the discussion is dedicated to (perceived) problems with C++ and D and other languages, but awfully light on details about Jai.
I’m pretty sure the beta targets the devs that are actually following Jonathan Blow on twitch. If you were never interested enough in watching, then the language is probably not for you at the moment.
I think it’s probably still a little too soon for that. The closed beta is definitely a sign that Jai’s definition is mostly solid, but I have no doubt that there will be changes to the compiler (maybe even the language?) in response to the feedback they get. Given Jonathon Blow’s habit of not committing to deadlines until something is really ready, I would continue to consider Jai to be unavailable until a publicly available version is announced.
You are confusing languages with implementations. Just because somebody has written a compiler, and claims that the compiler takes the Jai language as input, does not mean that the Jai language exists. We must not let private projects appear to be community projects.
You could make this same complaint of Zig, for what it’s worth. Zig isn’t specified yet, and it’s not clear whether a given hunk of code is valid Zig. (Although recently this attempt at specifying Zig seems to be quite good.)
I disagree strongly. If I would take your definition to heart then technically you’d be correct because the syntax is not definitive and the data model will never get what you might call a specification. But, I’m pretty sure that the Jai compiler will forever have only one implementation, the one released by Jonathan Blow. Even if he ever puts the source code under a permissive license, I doubt he will have the patience to deal with pull requests from contributors.
Despite all of that the Jai programming language can be used now and it produces binaries that run on multiple OSes, so for this specific case I would say that your statement makes a distinction without any difference. Adding the bold claim that a language that is not a community project is not a real language somehow is a level of pedantry that takes it even more over the top. :)
While I absolutely agree with the sentiment here, I do think it’s worth pointing out that the difference is not as stark as one might initially think. Access to Jai is limited because Jonathon Blow wants the language to be “right the first time”. Thus he discusses its features freely, but people can’t actually use it. You may believe this approach is smart or stupid, but it’s worth noting that while Zig is freely available, it is still at version 0.x. I’ve been able to use Zig to write small programs, but most of those programs no longer work because the current compiler (and/or standard library) has changed since then.
In other words, the distinction between the two is less qualitative and more just one of management.
The approach is demonstrably wrong. Nothing is correct the first time except for formal proofs, and there is no evidence that Blow is formally proving Jai’s behavior in either the implementation or specification. Jai will probably have minor revisions.
The way you’re commenting here makes me believe that you had no real contact with the work that’s being done on Jai and you only speak generalities about programming language design from the perspective of someone that enjoys the theory more than the rest. Jonathan Blow is probably the antithesis of that, and from what I’ve seen he abhors theory over pragmatism.
This is a language that’s been in the works for about 7-8 years now, there’s nothing “first time” about the compiler at this moment, and it will be even less so at release time, whenever that will be.
I’m very interested to see the follow-up to this, thanks! I think it is very much needed to both have a) an outside view of Jai in a non-trivial project (especially a game, which is what Jai is literally made for), and b) more views of nontrivial projects that are using safer-but-not-Rust-Safe languages like Jai and Zig.
Also, recording game state and inputs and using them to verify changes is awesome and from what I’ve heard from talking to pro’s it is basically essential in a high-powered gamedev project. Especially since once you can dump and re-play game state it is much easier to make debuggers and editors as well. It gives an amount of debuggability for complex interactive systems that lots of domains can only dream of. <3
For example, I believe a big chunk of these vulnerabilities would not exist if C and C++ (the most prominent “unsafe” languages for sure) simply didn’t have zero-terminated string, initialized values by default, had a proper pointer+length type thus replacing 90% of pointer arithmetic with easily bounds-checkable code, and had established a culture that discouraged the prevalent ad-hoc style of memory management.
I’ve been wondering about this lately: suppose we took C as-is and added the concept of slices (pointer + length) that are bound-checked and the program aborts if there’s an invalid access—would C programs be meaningfully more safe? Not perfect, obviously, but better? If anyone has insights or experience, I’d love to hear from you.
As a D fan, I agree with many of the criticisms, however, several I want to respond to:
-lowmemcompiler flag. This will make the frontend slower, but if you’re already burning 80% of your runtime in LLVM optimization routines, it probably has no perceptible effect on build speed.static foreach, are you trying to chainifstatements withstatic foreach? That fundamentally cannot work in any compiler that treatselseas syntactically part of theifcompound statement, which is basically every C-like language. Just put areturn/continue/breakat the end of eachifand chain regularifstatements.That said, you’re totally right about the codegen and debug info bugs, and about D not simply being “better C++”. (Also, named parameters should be coming any year now!) However, my prediction is that if you move to a smaller language, you will only encounter a greater volume of issues.
edit: I’m working on my own language btw ( https://neat-lang.github.io ), and I can confirm two things: unrestricted macros are pretty great, and so are named parameters. It’s very hard to go back to a language without them.
Rust people love to blame LLVM too, but it’s kind of a moot point. First, the user doesn’t care whether it’s slow because of LLVM or D. Second, LLVM is doing at least half of the work involved in compiling, if not more, so it’s unreasonable to expect it to take less time than the frontend. Third, Clang also uses LLVM, and compile times are nowhere near as bad. At Mozilla when they started adding Rust code into Firefox, that small fraction of the code took up a much larger portion of the total compile time–the vast majority of the code is still C++.
Well, if you can switch to the gcc-based gdc or the independent-backend dmd and see an improvement, it is interesting to note. Though there’s a few challenges with those (gdc has no binary build for Windows so you have to build it yourself right now, and dmd has some bugs and poor optimizations compared to the others) it still shows you might have other options.
Clang is literally the frontend whose specific usage patterns LLVM is optimized around though. “Clang is fast” doesn’t equate to “LLVM is fast”, only “LLVM is fast(ish) if you use it precisely like Clang uses it.” (Which is, of course, not documented.)
Anyway I agree that this is a problem for ldc, but I still feel it’s worth emphasizing to clarify that it’s not a language issue.
It is a language issue if your frontend uses LLVM, but uses it sub-optimally. I agree 100% that it isn’t fair, but the user doesn’t care. They’ll just switch to a different language.
It’s an issue for the language, not an issue with the language.
It becomes an issue with the language when there’s not many other options for compilers. Especially in Rust’s case.
Very interesting read, thanks. I tried to find more specific information about Jai, but only found https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md. Is there a language specification somewhere? It’s particularly interesting how a compile time decrease of 10-100x is possible if the developer at the same time is “able to do anything during compilation”.
The primer is quite obsolete at this point and the language creator doesn’t have any public documentation as far as I know. The only places where you can see fresh information is on Jon’s twitch stream and probably on the streams of some of the other devs or people having access to the beta.
Thanks. I wonder how then the author of the posted article is able to migrate a 40kSLOC project to Jai. How would he know about the language details and where would he get the compiler and debugger from?
From the article:
Most of the people that were interested in the language I think followed Jon’s work for a while so they could have been exposed to how it works. He is working at the same time on the compiler and a game plus dev tools developed in the language, which he streams on and off.
However for someone completely new, the beta comes with extensive examples extensively commented by the author, so most of the features are detailed there.
I don’t know much about Jai, but I can tell you this is a real issue with D. D compiles quite quickly if you keep to basic code - I have some old (smaller) games that compile in a fraction of a second (~30ms), and even my larger things like a 50k line web server with integrated gui tend to compile in ~2.5 seconds. Not bad at all…
At the same time, I can write a 100 line program that will take a minute to compile. Or more. Why? Because it does a bunch of compile time execution. So when you start running code at compile time, the number of lines of the project stops mattering. The number of lines stops mattering when you can run loops! And moreover, the way you write the compile time function can radically change it. D’s CTFE engine, for example, is very slow when doing large string appends in a loop. Similar code, rewritten to pre-allocate the string and copy chunks into it, which takes more lines of code and loops through the data twice (once to determine length, second to copy it over), can run 10x faster! (Such tricks sometimes help normal code too, but it isn’t as dramatic as here due to the ctfe engine’s own quirks of implementation)
So I’m very skeptical of any claim of language compile speed performance with D, and I expect Jai is the same, since it so heavily depends on how you do the compile time things. (to be fair to the critics, one of the selling points is that you can do this cool compile time function call stuff, so then people get excited and use it heavily… then it kills the build speeds. but if you advertise both it is fair for people to expect you can have both. and you can… but only to an extent and only if you take care to implement it well. You might still end up with a pre-build step that caches the result file and you can do that with C++ too.)
I assume this code is run by a kind of interpreter integrated with the compiler; if so the same performance issues apply as with every interpreter; there are ways to write faster interpreters, up to using JIT compilation; I didn’t have a look at the specific D compiler, so I don’t know what it actually does with compile time execution.
It is an AST interpreter that copies nodes any time a value is mutated. It is about the worst possible thing you can imagine that still manages to somehow work (I suspect this is why the OP saw gigabytes of ram uses as well as the long builds). You can definitely compare different implementations of those, and D’s can be significantly better in theory, but the point stands that comparing compile time per line of code is not a useful metric once you introduce any time of compile time function calls that can occur on one of those lines.
YouTube search “Jonathan blow language” , e.g
https://www.google.com/url?sa=t&source=web&rct=j&url=https://m.youtube.com/watch%3Fv%3DTH9VCN6UkyQ&ved=2ahUKEwjaz7X-v8T7AhVOHkQIHUpuBvcQwqsBegQICxAF&usg=AOvVaw1jFkoB7Qcd75L2erAB3lAo
Thanks, I already watched this talk from 2014, but I prefer a specification document and a compiler to make practical experiments.
Don’t bother. As far as I can tell, the Jai “team” doesn’t really want you touching their stuff.
Amen. I’m ending my current gig because of this, actually. Too few programmers have a taste for simplicity, let alone the will to achieve it. I personally find it alarming that I am consistently the most powerful simplifier in the room, just because I happen to hold the same Philosophy of Software Design as John Ousterhout. (Great book by the way.)
Dijkstra warned us long ago, complexity is supposed to be the bane of our existence. Yet most people seem either blind or resigned. And so we keep piling cruft on top of cruft. If I ever quit this job this will be the main cause.
Jai sounds a lot like zig. Any reason to choose one over the other?
Zig actually exists, and you can confirm that to yourself. Jai does not have a publicly available compiler.
That’s a pretty uncharitable comment. Jai exists but is not ready for public release, it’s as simple as that.
I didn’t mean it literally doesn’t exist, I just mean you can’t use it, so it might as well not - you can’t use it.
I think at the current time anyone that can be bothered to mail Jon with a small blurb about wanting to get into the beta can get in. At one point people had invitations available to give others.
Thanks for sharing, I was unaware there were copies of the compiler out there people are using build Jai programs. I’ll stop spreading misinformation.
I kind of have to agree with WilhelmVonWeiner on this. That’s quite a barrier to entry compared to other languages, especially for one with a bunch of kruft from C and C++.
Are there any public projects written in Jai demonstrating why I’d jump through hoops to use it? The list in the blog reads like they copy/pasted marketing fluff, and the majority of the discussion is dedicated to (perceived) problems with C++ and D and other languages, but awfully light on details about Jai.
I’m pretty sure the beta targets the devs that are actually following Jonathan Blow on twitch. If you were never interested enough in watching, then the language is probably not for you at the moment.
I think it’s probably still a little too soon for that. The closed beta is definitely a sign that Jai’s definition is mostly solid, but I have no doubt that there will be changes to the compiler (maybe even the language?) in response to the feedback they get. Given Jonathon Blow’s habit of not committing to deadlines until something is really ready, I would continue to consider Jai to be unavailable until a publicly available version is announced.
You are confusing languages with implementations. Just because somebody has written a compiler, and claims that the compiler takes the Jai language as input, does not mean that the Jai language exists. We must not let private projects appear to be community projects.
You could make this same complaint of Zig, for what it’s worth. Zig isn’t specified yet, and it’s not clear whether a given hunk of code is valid Zig. (Although recently this attempt at specifying Zig seems to be quite good.)
I disagree strongly. If I would take your definition to heart then technically you’d be correct because the syntax is not definitive and the data model will never get what you might call a specification. But, I’m pretty sure that the Jai compiler will forever have only one implementation, the one released by Jonathan Blow. Even if he ever puts the source code under a permissive license, I doubt he will have the patience to deal with pull requests from contributors.
Despite all of that the Jai programming language can be used now and it produces binaries that run on multiple OSes, so for this specific case I would say that your statement makes a distinction without any difference. Adding the bold claim that a language that is not a community project is not a real language somehow is a level of pedantry that takes it even more over the top. :)
So the question remains, why he at all made videos/presentations to announce Jai, starting nearly ten years ago; if you want public feedback for a language, the first thing you publish is a specification or “language report” (as e.g. Wirth calls it). This document specifies purpose, syntax and semantics of a language. Examples: https://people.inf.ethz.ch/wirth/Oberon/Oberon07.Report.pdf, https://github.com/oberon-lang/specification/blob/master/The_Programming_Language_Oberon%2B.adoc, http://software.rochus-keller.ch/busy_spec.html
If there is no specification, people have to make assumptions on the language based on some examples, and the feedback is close to useless.
While I absolutely agree with the sentiment here, I do think it’s worth pointing out that the difference is not as stark as one might initially think. Access to Jai is limited because Jonathon Blow wants the language to be “right the first time”. Thus he discusses its features freely, but people can’t actually use it. You may believe this approach is smart or stupid, but it’s worth noting that while Zig is freely available, it is still at version 0.x. I’ve been able to use Zig to write small programs, but most of those programs no longer work because the current compiler (and/or standard library) has changed since then.
In other words, the distinction between the two is less qualitative and more just one of management.
The approach is demonstrably wrong. Nothing is correct the first time except for formal proofs, and there is no evidence that Blow is formally proving Jai’s behavior in either the implementation or specification. Jai will probably have minor revisions.
The way you’re commenting here makes me believe that you had no real contact with the work that’s being done on Jai and you only speak generalities about programming language design from the perspective of someone that enjoys the theory more than the rest. Jonathan Blow is probably the antithesis of that, and from what I’ve seen he abhors theory over pragmatism.
This is a language that’s been in the works for about 7-8 years now, there’s nothing “first time” about the compiler at this moment, and it will be even less so at release time, whenever that will be.
Relax. It is not your fault that Blow has not released their code.
I’m very interested to see the follow-up to this, thanks! I think it is very much needed to both have a) an outside view of Jai in a non-trivial project (especially a game, which is what Jai is literally made for), and b) more views of nontrivial projects that are using safer-but-not-Rust-Safe languages like Jai and Zig.
Also, recording game state and inputs and using them to verify changes is awesome and from what I’ve heard from talking to pro’s it is basically essential in a high-powered gamedev project. Especially since once you can dump and re-play game state it is much easier to make debuggers and editors as well. It gives an amount of debuggability for complex interactive systems that lots of domains can only dream of. <3
I’ve been wondering about this lately: suppose we took C as-is and added the concept of slices (pointer + length) that are bound-checked and the program aborts if there’s an invalid access—would C programs be meaningfully more safe? Not perfect, obviously, but better? If anyone has insights or experience, I’d love to hear from you.
https://github.com/microsoft/checkedc