Thanks. I actually think effect systems are going to find use in the future in many languages.
I was thinking of describing eyg as a strongly typed Lua for a while. And dropping the structural editor, but for now I’m keen to keep both experiments going. even if it makes the language a bit more niche in the meantime.
When will people learn? 30 years of Symonyi’s work, the countless hours put into IntelliJ’s MPS and Eclipse’s XText, half of Bracha’s thankless toiling, all ended up being ignored. Structural editing needs to operate “in the background” the way treesitter does. People are used to editing text. Let people mangle the text how they see fit, and just update your model when it parses. I’m the first in line to rail against plain text as the storage format and API of a programming language, but it is the best user interface for actually writing code when you’re trying to get ideas out of your head into the computer. It doesn’t have to be plain monospaced bollocks though, let people format it, set their own colours for bindings, insert tables, put images in comments, etc.
No one ever made progress by giving people more of what they already had.
it is the best user interface for actually writing code when you’re trying to get ideas out of your head
It is easier to learn text. And you already know text. That doesn’t mean it is the best ui. I also maintain that there haven’t been any really good structural editors yet, though fructure and hazel look interesting—real advantage comes from large-scale manipulation and analysis (incl. programmatic—take semgrep-type as one random example, but can be much more powerful and simpler; also smalltalk class browser). Text-based interfaces may converge on something in the ballpark of what you can do with pure structure eventually, but at the cost of much pointless complexity and—still—some power (eg presentation types for ast nodes), and giving back nothing in exchange but familiarity and a bit of ease of learning. If what you want is to get ideas out of your head, code may not be the most appropriate medium—that’s fine. I often work out ideas with pen and paper (NB. not text either!).
I don’t think that maintaining bidirectional mapping between text and AST is complex, it’s just little known how to do that. If you factor the cost of associated tooling which just works with text and needs to be re-implemented for each non-text language (editor, vcs, online forge), it seems that working with text is a couple of orders of magnitude less complex?
My impression is that JetBrains basically figured out how to do syntactic and semantic aware coding in early aughts, and then everyone ignored this cluster of ideas for practical usage until LSP and Tree-sitter came along.
ASTs are simple and hierarchical. If the goal is to support complex, graphical relationships, this becomes much more challenging. You would at the very least need weak pointers and in-band signalling—bad things, both.
editor, vcs, online forge
Editor I have argued is simplified by operating only on structure; unless you disagree, leave that aside. The basic function of a vcs can work just the same on opaque files (arbitrary serialisation format—necessary, but incidental and trivial); patch/diff you want to be language-aware anyway. Online forge I don’t find too interesting—in particular, its functions should be subsumed by the editing environment; e.g., you should be able to have strong pointers between tickets and code elements—but for the ‘online’ part, you may perhaps compile your editor to javascript or wasm.
More broadly, conceptually unintegrated (cf brooks) programming environments are bad; generic tooling is going to be suboptimal anyway. The preponderance of independent, disconnected programming languages and environments is an unfortunate accident that would not have been necessary given better abstractions, though it may not have been avoidable (see gabriel): the hegemony of mediocre unix meant that it was necessary to make better programming languages on top of it, but they could only interoperate at the level of its abstractions, which were limited to flat, inexpressive text, which was of necessity somewhat removed from the far richer and more useful structural abstractions used within those languages. A better-abstracted and better-constructed system would admit structural interoperation.
Note that Sophistifunk is talking about writing code while also maintaining a model which isn’t plain text. In that scenario, structured manipulation across the code base is still possible via the model alongside having localised “freeform” text entry. Personally I lean towards your view, but the freeform entry direction is also worth exploring I think.
writing code while also maintaining a model which isn’t plain text
Yes—‘projectional editing’. But it becomes increasingly cumbersome to maintain the textual mapping—hence ‘much pointless complexity’—if you want to go both ways. If it’s a one-way mapping—write a new function as text, compile it to structure and integrate it with the running system, and manipulate it thus thereafter—then fine, I suppose, but also kind of pointless imo. I’ll refer also to a brief exchange on the matter I had with geoff langdale.
Taking this further: I find it completely reasonable that a good structured editor might support complex transactions which can not easily be decomposed into structurally and semantically coherent small-steps. But that does not at all mean that textual manipulation is the best way to implement those transactions.
What I mean is that you maintain a structured model of the code that is used to drive the rest of your process, and also to derive useful information to help the human editing the code, but don’t restrict the user to operations that are atomic with respect to maintaining a correct model. You can allow the user all the wonderful model-aware operations should they chose to trigger one, but don’t force them to do so. If they just want to mess with the text let them do it, and simply maintain your last-known-good model of the text while they do so. When the text once again becomes parseable, then you update your model and re-run any dependent inspections / start showing errors / whatever you would normally update if they had made the change wholesale as an atomic operation.
In structural editors where it’s touted that the user “can’t put the source into an invalid state” it means that the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be, which can and often does present a serious extra cognitive load on the user, as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition.
Hello I am making a multi-language structural editor.
the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be
Indeed, while text can be edited with “left, right, up, down, backspace, newline, characters”, trees have more structure and so require more basic commands. Mapping it out, having a set of commands that’s about as powerful as the basic Vim commands seems to require roughly the same number of keys, with related but tree-y functionality. Here’s a sketch for a Vim-like key map (just the first set; everything after “——” is old notes):
There’s one thing you need to know while editing that isn’t present in the key map: when inserting a node, you need to say what node type it is (e.g. “+” or “function” or “if”). Our plan is that there’s a single key for each (e.g. “+”, “f”, “i”) that is shown on screen when you go to insert, to make them discoverable. This should actually increase discoverability compared to a text editor, when editing a language you’re not very familiar with.
as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few? I’m very curious what obstacles there are.
One important thing is that — at least in the structural editor I’m making — is that there can be “holes” in the document. For example, showing a hole as ?, an if/else whose condition hasn’t been filled in would be if (?) { x += 1; } else { x -= 1; }. So there’s a little bit of allowed syntactic invalidity that might make it easier to traverse between syntactically valid states.
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few?
Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes. Every single time that’s happened, you’d have had to stop and re-think not just “what do I want the code to look like when I’m done with this edit?” but also “how do I get to where I want to be using only the allowable operations within this editor?”. And perhaps when you’re the one building the structural editor this seems like it’s not a burden, but as somebody who needs to do stuff like this all the time, I can assure you it’s not. I’m not interested in adding any barriers to my job, I want tools to help me, to work for me, not constrict me in order to make their own implementation easier.
Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes.
Can you be more specific, like with actual code examples? Because I move things around all the time, and the program tends to stay syntactically valid the whole time. Or rather, it sometimes becomes invalid but only because of silly punctuation things that a structural editor would take care of. Some examples:
Example 1: swapping two list elems in JSON by cutting and pasting.
In a structural editor, you cut the third element and paste it before the first element, then cut the first element and paste it after the second element, and at all five of these steps the commas are correct.
Example 2: wrapping some stuff in a function.
Say you realize that some code ought to be inside a function so you can re-use it, and you write half the function but don’t finish it:
fn big_function() {
...
forward_map.insert(key, val);
reverse_map.insert(val, key);
...
}
-->
// syntactically invalid because key doesn't have a type
fn helper_function(key) {
forward_map.insert(key, val);
reverse_map.insert(val, key);
}
fn big_function() {
...
}
with holes, this is actually “syntactically valid”:
I like to keep my program syntactically valid, because my auto-formatter only works when the program parses. So it’s possible that you edit differently than I do, going through more invalid states. If so, I’m very curious what states those are.
A wise man once said that critics are your best friends because, unlike your fans, they help you improve your work with constructive feedback. Won’t you be my friend?
Uhu, I also tend to think that anything you can do in a structural editor, you can do in a normal editor as well, plus you’ll get all nice pure-text editing features (e.g, multiple cursors) for free.
The only issue here is that for this to work you also need a parser which doesn’t produce a train wreck of an AST if there’s a missing semicolon, and there’s frustratingly little information on how to do that. https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html explains the trick.
All of them? Speaking for at least one person I like editing the structure directly.
Probably I should be clearer. I’m building the language for myself and (if there are any) like minded people. The project (language & editor) are optimised for exploring ground that is new and interesting to me and not for mass appeal. For potential mainstream appeal I point people at https://gleam.run/ anyway.
Based on the source this looks to be written in gleam, very cool! Any resources on actually playing around with the language locally and trying out some simple programs?
For now it always runs inside the browser. so using the version at eyg.run has the same capabilities as running that webpage locally would have. There are instructions for starting the site locally here https://github.com/CrowdHailer/eyg-lang/tree/main/website.
Project homepage is here https://eyg.run/
@crowdhailer
I like the portability and effect system. Sending code over the wire made me think of Telescript. It’s also a nice-looking, summary page.
Thanks. I actually think effect systems are going to find use in the future in many languages. I was thinking of describing eyg as a strongly typed Lua for a while. And dropping the structural editor, but for now I’m keen to keep both experiments going. even if it makes the language a bit more niche in the meantime.
When will people learn? 30 years of Symonyi’s work, the countless hours put into IntelliJ’s MPS and Eclipse’s XText, half of Bracha’s thankless toiling, all ended up being ignored. Structural editing needs to operate “in the background” the way treesitter does. People are used to editing text. Let people mangle the text how they see fit, and just update your model when it parses. I’m the first in line to rail against plain text as the storage format and API of a programming language, but it is the best user interface for actually writing code when you’re trying to get ideas out of your head into the computer. It doesn’t have to be plain monospaced bollocks though, let people format it, set their own colours for bindings, insert tables, put images in comments, etc.
No one ever made progress by giving people more of what they already had.
It is easier to learn text. And you already know text. That doesn’t mean it is the best ui. I also maintain that there haven’t been any really good structural editors yet, though fructure and hazel look interesting—real advantage comes from large-scale manipulation and analysis (incl. programmatic—take semgrep-type as one random example, but can be much more powerful and simpler; also smalltalk class browser). Text-based interfaces may converge on something in the ballpark of what you can do with pure structure eventually, but at the cost of much pointless complexity and—still—some power (eg presentation types for ast nodes), and giving back nothing in exchange but familiarity and a bit of ease of learning. If what you want is to get ideas out of your head, code may not be the most appropriate medium—that’s fine. I often work out ideas with pen and paper (NB. not text either!).
I don’t think that maintaining bidirectional mapping between text and AST is complex, it’s just little known how to do that. If you factor the cost of associated tooling which just works with text and needs to be re-implemented for each non-text language (editor, vcs, online forge), it seems that working with text is a couple of orders of magnitude less complex?
My impression is that JetBrains basically figured out how to do syntactic and semantic aware coding in early aughts, and then everyone ignored this cluster of ideas for practical usage until LSP and Tree-sitter came along.
ASTs are simple and hierarchical. If the goal is to support complex, graphical relationships, this becomes much more challenging. You would at the very least need weak pointers and in-band signalling—bad things, both.
Editor I have argued is simplified by operating only on structure; unless you disagree, leave that aside. The basic function of a vcs can work just the same on opaque files (arbitrary serialisation format—necessary, but incidental and trivial); patch/diff you want to be language-aware anyway. Online forge I don’t find too interesting—in particular, its functions should be subsumed by the editing environment; e.g., you should be able to have strong pointers between tickets and code elements—but for the ‘online’ part, you may perhaps compile your editor to javascript or wasm.
More broadly, conceptually unintegrated (cf brooks) programming environments are bad; generic tooling is going to be suboptimal anyway. The preponderance of independent, disconnected programming languages and environments is an unfortunate accident that would not have been necessary given better abstractions, though it may not have been avoidable (see gabriel): the hegemony of mediocre unix meant that it was necessary to make better programming languages on top of it, but they could only interoperate at the level of its abstractions, which were limited to flat, inexpressive text, which was of necessity somewhat removed from the far richer and more useful structural abstractions used within those languages. A better-abstracted and better-constructed system would admit structural interoperation.
Note that Sophistifunk is talking about writing code while also maintaining a model which isn’t plain text. In that scenario, structured manipulation across the code base is still possible via the model alongside having localised “freeform” text entry. Personally I lean towards your view, but the freeform entry direction is also worth exploring I think.
Yes—‘projectional editing’. But it becomes increasingly cumbersome to maintain the textual mapping—hence ‘much pointless complexity’—if you want to go both ways. If it’s a one-way mapping—write a new function as text, compile it to structure and integrate it with the running system, and manipulate it thus thereafter—then fine, I suppose, but also kind of pointless imo. I’ll refer also to a brief exchange on the matter I had with geoff langdale.
Taking this further: I find it completely reasonable that a good structured editor might support complex transactions which can not easily be decomposed into structurally and semantically coherent small-steps. But that does not at all mean that textual manipulation is the best way to implement those transactions.
I’d love it if I learned more about this right now :)
I’ve learned about structural editing recently, and I’d love pointers to good resources.
What do you mean by this?
What I mean is that you maintain a structured model of the code that is used to drive the rest of your process, and also to derive useful information to help the human editing the code, but don’t restrict the user to operations that are atomic with respect to maintaining a correct model. You can allow the user all the wonderful model-aware operations should they chose to trigger one, but don’t force them to do so. If they just want to mess with the text let them do it, and simply maintain your last-known-good model of the text while they do so. When the text once again becomes parseable, then you update your model and re-run any dependent inspections / start showing errors / whatever you would normally update if they had made the change wholesale as an atomic operation.
In structural editors where it’s touted that the user “can’t put the source into an invalid state” it means that the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be, which can and often does present a serious extra cognitive load on the user, as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition.
Hello I am making a multi-language structural editor.
Indeed, while text can be edited with “left, right, up, down, backspace, newline, characters”, trees have more structure and so require more basic commands. Mapping it out, having a set of commands that’s about as powerful as the basic Vim commands seems to require roughly the same number of keys, with related but tree-y functionality. Here’s a sketch for a Vim-like key map (just the first set; everything after “——” is old notes):
https://github.com/justinpombrio/synless/blob/master/doc/commands.txt
There’s one thing you need to know while editing that isn’t present in the key map: when inserting a node, you need to say what node type it is (e.g. “+” or “function” or “if”). Our plan is that there’s a single key for each (e.g. “+”, “f”, “i”) that is shown on screen when you go to insert, to make them discoverable. This should actually increase discoverability compared to a text editor, when editing a language you’re not very familiar with.
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few? I’m very curious what obstacles there are.
One important thing is that — at least in the structural editor I’m making — is that there can be “holes” in the document. For example, showing a hole as
?, an if/else whose condition hasn’t been filled in would beif (?) { x += 1; } else { x -= 1; }. So there’s a little bit of allowed syntactic invalidity that might make it easier to traverse between syntactically valid states.Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes. Every single time that’s happened, you’d have had to stop and re-think not just “what do I want the code to look like when I’m done with this edit?” but also “how do I get to where I want to be using only the allowable operations within this editor?”. And perhaps when you’re the one building the structural editor this seems like it’s not a burden, but as somebody who needs to do stuff like this all the time, I can assure you it’s not. I’m not interested in adding any barriers to my job, I want tools to help me, to work for me, not constrict me in order to make their own implementation easier.
Can you be more specific, like with actual code examples? Because I move things around all the time, and the program tends to stay syntactically valid the whole time. Or rather, it sometimes becomes invalid but only because of silly punctuation things that a structural editor would take care of. Some examples:
Example 1: swapping two list elems in JSON by cutting and pasting.
In a text editor:
In a structural editor, you cut the third element and paste it before the first element, then cut the first element and paste it after the second element, and at all five of these steps the commas are correct.
Example 2: wrapping some stuff in a function.
Say you realize that some code ought to be inside a function so you can re-use it, and you write half the function but don’t finish it:
with holes, this is actually “syntactically valid”:
I like to keep my program syntactically valid, because my auto-formatter only works when the program parses. So it’s possible that you edit differently than I do, going through more invalid states. If so, I’m very curious what states those are.
A wise man once said that critics are your best friends because, unlike your fans, they help you improve your work with constructive feedback. Won’t you be my friend?
Check out hazel and associated papers!
Hazel is a great project.
Another interesting one to look at is https://tylr.fun/ very pretty
Uhu, I also tend to think that anything you can do in a structural editor, you can do in a normal editor as well, plus you’ll get all nice pure-text editing features (e.g, multiple cursors) for free.
The only issue here is that for this to work you also need a parser which doesn’t produce a train wreck of an AST if there’s a missing semicolon, and there’s frustratingly little information on how to do that. https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html explains the trick.
All of them? Speaking for at least one person I like editing the structure directly.
Probably I should be clearer. I’m building the language for myself and (if there are any) like minded people. The project (language & editor) are optimised for exploring ground that is new and interesting to me and not for mass appeal. For potential mainstream appeal I point people at https://gleam.run/ anyway.
This point is made slightly clearer in the project README. https://github.com/CrowdHailer/eyg-lang/
Based on the source this looks to be written in gleam, very cool! Any resources on actually playing around with the language locally and trying out some simple programs?
For now it always runs inside the browser. so using the version at eyg.run has the same capabilities as running that webpage locally would have. There are instructions for starting the site locally here https://github.com/CrowdHailer/eyg-lang/tree/main/website.
It is possible to run the cli locally using
gleam run cliwhich will hit this part of the program https://github.com/CrowdHailer/eyg-lang/blob/main/eyg/src/eyg.gleam#L11-L13 But that is not documented. probably what I need to work on soon but for now I’m focusing on making the editor good.Great to see a project written in Gleam getting attention!