I’ve been thinking a lot lately about how deeply underutilized our visual cortex is for programming tasks, so this is good timing.
I think there’s a lot to learn about this from data visualization experts, but the example in the article with the bar-chart vs scatter-plot drives home just how difficult it can be to take a set of data and produce useful/interesting visualizations that communicate useful findings.
One thing that popped into my mind while reading this was of alternative uses for textual highlighting. There’s all kinds of information that could be useful when reading code, and using text color could be really useful - my first inclination was a sort of code “hotness” rating that highlights functions differently based on how widespread their use in the rest of the codebase is, or the number of times a particular function has been changed (a sort of “Chesterton’s fence” flag for functions with lots of covered edge-cases.)
One thing that popped into my mind while reading this was of alternative uses for textual highlighting.
I think this comment chimes with what I took away from the post. using the visual cortex more could be powerful and it we don’t have a better answer why not start from the text. I’ve been thinking about this while building EYG a language for a projectional editor. Technically any visualisation would be possible but the text version was the smoothest experience for me.
There are lot’s of different use of highlighting I have thought about.
One is to hover over an expression in the code and have it highlight that is needed to evaluate that value. or visa versa all the code that depends on that expression being evaluated.
Effects, I have an effect system and so all function calls can be highlighted by their effect. I had different colors for each effect and a dark mode node when the function was polymorphic in it’s effects.
More can be done with git integration. A heat map on how often the function changes would be interesting.
I was listening to your podcast with Richard Feldman, and I found it more insightful than I originally was anticipating. I forget exactly what though. I’ll have to go back and listen.
With the first two, it sounds like you kind of want a preview of how something is tied to all other parts of the code base. And because code relations are multidimensional, we don’t visualize it all at once. So perhaps like how smalltalk’s browser looks (emulated by apple’s column view), but much smaller, like a sparkline. Then you can tell how many other classes or methods in the system are affected and how far away they are from the current line of code.
I think I’d want some situational awareness of how changing one thing affects anything else in the codebase. I would think either sourcegraph or language servers would have more of this information in it, and might be available to provide this info in addition to git. It’s just that we haven’t hooked it up per a given use case while reading or writing code.
visual studio kinda does that: it inserts a “n references” line above each function, at least in C#. you can click it to see the references
it’s not a heatmap, so it takes up space in the editor, though. i think i’d want distinct colours for 0 and 1, and then a gradient for 2+. knowing zero/one/many is useful for refactoring, so distinguishing between them would be nice
I think you can apply code “hotness” for a variety of different definitions of hotness. It could be “last places where things were edited” or “places where there was a security vulnerability”. Or even a kind of “reverse observability”, where information from the production server is fed back into your IDE, so you can see which code paths are hot in real time.
I recall one of my professors working on a tool that showed (simulated) energy consumption with syntax highlighting, for low-power embedded applications. Almost literal hotness!
How about “Functions don’t kill people; side effects do”?
I liked the article (title aside) but wished there were some more details about how effects work.
You alluded to effects being part of a function’s type signature. It looks as though this is invisible in the source code; the compiler sees an effect being performed and annotates the function with it. If a function calls another that has an effect (without a “handle” clause) the compiler tags it with the effect too, and so on.
Is it possible to declare a function’s effects explicitly, so the compiler can issue an error if it calls something that has an undeclared effect? Otherwise it sounds like adding an effect to a lower level function “taints” everything above it in the call tree with that effect, which could have unpleasant consequences.
How effects work is at least a few different things.
How the runtime works, I think this is closest to conceptually what they are. And what I was trying to explain here.
How they are typed, not what I was explaining here but if so maybe I shouldn’t mention types at all in that case. Note effects can exist in untyped languages.
How effects can be handled, a further article I have in draft.
So it seems like I should try and write about the type system before getting on to handlers.
Not in EYG there is no possibility to declare any types they are only ever inferred. But in the general case effects end up in a function type signature so if you can annotate a function that annotation should include the effects.
The title of this article is, in my opinion, in poor taste. The article makes almost no mention of it, and I was expecting an article related to its title, and not an explanation of the author’s programming language’s implementation of algebraic effects (or at least a link between the two that was more than a couple of sentences.)
There also isn’t an explanation as to why it would be computers and not the program (or here, the programming language.) It’s an important and interesting discussion, but it’s used only to bait-and-switch; and in particular, by using a variant of the known sentence that removes all responsibility from the program’s authors.
I think you make a fair point here. I’ve been iterating on the post for a while and hadn’t paid attention to the fact the last edit removed any mention. The cliche that side effects might “launch the missiles” is referenced only in the images. There is also no point in an explainer article having slight reference to “in jokes”.
by using a variant of the known sentence that removes all responsibility from the program’s authors.
This is an interesting thought. I really only wanted to emphasize that programs only act on input and have no “real” knowledge of the outside world. I’m hadn’t been making commentary on the program authors responsibility. Though as you correctly indicate also an interesting discussion.
I will think of an alternative title. “Explaining algebraic effects” might be all that’s needed.
It would be better without the clickbait title. Maybe fewer would read it, but those who do will take it more seriously. There are genuine ethical issues involved in authoring software; let’s not cheapen them.
I listened to several talks from the “sprites” (name made up on a whim just now) at fossdem.
I’m very interested in finding out more about the early days and how it came into existence. It seems a genuinely interesting set of experiments and I hope to keep more up to date with what they do.
I really like how he considers things through the charts he shows (although I’d choose different elements and positions), especially the idea behind the problem space one at 2:45.
3:45’s language features chart evokes cool visualizations in the direction of https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.pdf (though his is just good vs. bad). 9:30’s version seems to have better axes, but keeping things in the same position baffles me. (I’m not sure how to interpret a dichotomy between indirection and abstraction either… indirection’s a mechanism to build abstractions or subordinate detail, no?)
His note on Gall’s law (complex working systems evolved from simple working systems):
start with predictable things and make them applicable, rather than giving you all the powerful features, because constraining powerful features is harder than extending predictable features
is quite insightful! Traditions within a (language) community form around shelling points, which endure by inertia as more features are added. When a language appears with a massive primitive set or no clear (best) way to do something (Unison, Flix, people flounder around and adoption doesn’t happen. There’s a clear relation to “single paradigm” (or PEP20) languages gradually branching out with feature bloat. (On the other hand… Perl? I’m not sure how true this is.)
Eat your greens - do things which are good for you, even if not immediately enjoyable or easy
Calling things with delayed benefits vegetables is a great turn of phrase. I will start saying EYG.
I disagree with his idea that Moore’s law enables more expressive, predictable paradigms because Forth & APL excelled with minimal resources (even if Lisp, Prolog, Haskell or Rust’s compiler require modern processing power.) Rust may have 0 cost abstractions, but some paradigms actually shift the paradigm!
Unfortunately, I’m close minded and hate the EYG (the language)’s syntax (inelegant notation constraining thought) although he presents well thought-out semantics.
Nothing in the program can actually know the limits of what side effects do
is a very interesting phrasing. I’ve always thought about enforcing invariants and protecting internal logic’s correctness but somehow never thought in the opposite direction, to protect the environment from a runaway program. When he later says “the outside world calls the continuation”, well huh, I suppose the environment does choose to start a program (besides SRE considerations like powering the computer running it etc.)
!capture’s cool. I like capture the AST and passing it to another environment! Hopefully some future OS lets us even transfer closures to other programs.
Thanks for the thoughtful comment.
To me indirection has negative connotations, though maybe to be clear I could say obfuscation instead.
That’s a fair point about Gall’s law not really playing out for Perl, probably a similar argument applies to languages like Ruby.
Syntax always gets strong reactions, I already know of one effort for someone to write their own syntax on top of the EYG AST.
Capture has been a really fun discovery I really hope to make a lot of it in some upcoming projects.
I’d also love to extend an invite to the BABLR community. We have a ton of alignment with your mission: we’re trying to take it the next step by offering an editor in the spirit of EYG’s that can work with any and every programming language.
Almost everyone says that. I’m sure I could do a better job of explaining, but also part of the problem is that it’s a new kind of thing – in a category of its own. I feel like it aught to be easiest to explain it to you though! It’s no more and no less than what is needed to make an EYG-ish editor experience for an arbitrary programming language, even a complicated real-world language like Javascript.
Like you I think that having a browser native experience gets code editing into a lot of exciting spaces that programming needs to be like TVs in homes and tablet computers in classrooms – even and especially places where code literacy is needed but installation of a native application would be a major barrier to access.
To build structural editing for an arbitrary language you need some kind of system of parsers to define the languages and some kind of universal node-tree representation which has the property of being able to snap nodes together as if they were lego bricks, just like the EYG editor does.
CSTML is the bottom layer of my infrastructure: the lego bricks. You could think of it as a reimagination of XML that learns from JSON and GraphQL to fix XML’s flaws. It has three big innovations in it:
Unlike XML, nodes have named relationships to each other which makes expressing syntax trees far more natural. You could imagine that in HTML and XML all nodes have an implicit relationship called “children” with their children, but CSTML is able to add fast random access via named relationships as a layer on top of the implicit ordered-children relationship
Like HTML, CSTML nodes can be said to have “inner text” embedded in them. This embedded text is the source code of the program as the parser saw it, and serves as the source of truth for semantics. Having a source of truth is very important because the process of parsing is going to denormalize this data by injecting it with a great deal of metadata – in particular the structure of the parse tree expressed as enclosing tags like <*BooleanLiteral> 'true' </>. Having inner text ensures that we are never at a loss for how to print the code the tree is meant to be printed, which is a very important property for being able to make editors. CSTML is not meant to be seen directly except by developers, rather it is meant to be a data layer which can encapsulate any state that might be produced by a powerful GUI tool.
Lastly CSTML has gaps. It permits you to declare that part of a tree is missing in a particular location. It does not ever permit invalid syntax trees (I do not wish to allow them in my editor), so allowing incomplete trees is the necessary escape hatch. A tree with a gap in it is, conceptually, a template. The smallest possible template is a syntax node, and lego brick with gaps into which other syntax nodes may be placed.
On top of CSTML is BABLR, the schema validation layer. It helps you narrow the infinite space of arbitrary valid CSTML down to just what should be valid according to the rules of your programming language. “Parse don’t validate,” is our philosophy, so once we’ve parsed nodes we make them immutable. Deeply immutable trees of syntax nodes lead to easy caching and another editor feature we want: the ability to hold history efficiently just by holding on to prior roots of the immutable data structure.
no less than what is needed to make an EYG-ish editor experience for an arbitrary programming language,
This works very well as an explanation for me. An admirable mission. The EYG editor was challenging and I was allowed to mold EYG the language to my purpose.
The definition of operating system in this article is so broad as to be meaningless. Words have meanings, and while the area of systems software on the whole is indeed quite nebulous, an operating system is a relatively specific subset that does not extend to Kubernetes or the a bytecode VM that runs as an application program.
The edges are still fuzzy of course, but I would expect at least a few things to be true generally:
an operating system takes over sole control of CPUs in a machine, from boot onwards
the operating system directly interfaces with features of the CPU for switching tasks and scheduling things
if the operating system was not present, or has broken somehow, the machine would be a brick
an operating system manages hardware resources like memory and network interfaces and storage devices, virtualising those resources and providing a mixture of sharing and isolation depending on policy choices the user makes
the operating system generally runs at some elevated privilege level beyond regular user software
most of the software that’s running on the machine is not the operating system, it’s application software that does something the user actually cares about
Kubernetes is ultimately application software that runs on top, using OS facilities and providing complex orchestration to co-ordinate a fleet of machines.
If your software uses the Berkeley sockets API to co-ordinate with other instances of itself over a regular TCP/IP network, it’s not the operating system.
This is, honestly, the reason that the typical definition of “operating system” doesn’t really make sense anymore. Layers of operating systems. Will the real operating system please stand up?
The hypervisor is the OS for the hardware, Linux is then the OS for the guest, virtual machine… in the traditional sense.
Like I said, the edges are a bit fuzzy, but there have been virtualised machines with hypervisors for a very long time, and I don’t think it defrays the meaning of the term. Generally the OS software that runs in a virtual machine is capable of operating the whole hardware, it’s just been constrained to a partition of sorts. That constrained slice could just as well have been made by a hardware partitioning technology instead of a software one. I think what makes the hypervisor and virtual machine concept so legible is that the idea of the OS inside retains its meaning.
I’m hugely sceptical of the value of unikernels, but it’s pretty clear that the library or framework you jam into an image alongside whatever application payload is an operating system. In this case it seems most correct to say that LING is a toolkit for making a complete operating system with an Erlang-based application payload included in the image, rather than that “Erlang is an operating system”.
I’m not sure “it’s pretty clear that the library for framework you jam into the image alongside whatever application payload is an operating system” is so clear. If that’s the case, then Linux running on Xen isn’t an operating system (based on your definition above), only the implementation of the Xen ABI is, which is just a small part of Linux as a whole.
The situation is more visibly clear with Erlang on Xen, which is why I brought that up specifically. Erlang has its own process isolation model, scheduler, and memory management. Typically, it uses the abstraction that the OS provides to provide these things, and on Xen, presumably Erlang’s implementation of these things is of much greater importance, in terms of how effective the underlying hardware is used, since the Xen ABI implementation is likely much less primitive than Linux itself.
But sure, all of this might be “fuzzy edges” and “fuzzy middles,” too, me thinks.
But now you have to classify Erlang as an operating system
I initially reacted negatively to this based on instinct, but have changed my mind especially because of this description:
Erlang has its own process isolation model, scheduler, and memory management.
If Erlang - or to be more specific, the Erlang VM - isn’t an operating system (especially with LING attached), then what do you call the software that ran Lisp Machines?
If Erlang - or to be more specific, the Erlang VM - isn’t an operating system (especially with LING attached), then what do you call the software that ran Lisp Machines?
To be clear, I’m merely arguing that the definition of an “operating system” is no longer as simple as it once was.
The Lisp Machines ran custom hardware and were designed to run Lisp efficiently. I assume the operating system was almost completely written in Lisp, minus likely, a small set of “assembly” routines to bootstrap a basic reader and compiler, to bring up a more capable system.
I forget the exact quote but it went something like “a programming language consists of all the bits they forgot to include in the operating system” so yeah, why not?
what’s your take on unikernels then? specifically thinking of mirage, although I don’t have a lot of knowledge in that domain, just read about it a while ago.
If your software uses the Berkeley sockets API to co-ordinate with other instances of itself over a regular TCP/IP network, it’s not the operating system.
This is possibly a way to define operating systems, and I think it’s one a lot of people understand.
I think I want a general term for a system that accepts code to run. orchestration tool is a contender, but that to me does feel more focused on aspects that make up a clustered system.
an operating system takes over sole control of CPUs in a machine, from boot onwards
I guess at that point that you would consider DCOS misnamed https://dcos.io/
I guess at that point that you would consider DCOS misnamed https://dcos.io/
This is called an “OS” because “distributed orchestration system” isn’t as recognizable from a marketing perspective. We can’t allow technical definitions to be redefined for marketing purposes.
Imagine the insanity of someone selling square roots of numbers, but marketing decided to call them qwijibos instead. Absolutely ridiculous! They are square roots!
The web site is https://eyg.run. I couldn’t find anything about a source code repository, or how to install eyg on my Linux machine so that I can try it out and use it. It seems that an implementation exists, but nothing has been released yet. It looks interesting, somebody should post to Lobste.rs when there is a release, then I will spend time looking at it. Didn’t watch the video, sorry.
This is correct, although it probably still doesn’t have what you’re looking for regarding install instructions.
The editor and running that exist on the web page are THE only up to date editor and latest runner.
Working in the browser is the environment it’s for.
Not strictly related to the submission, but: when do people actually listen to podcasts? I can’t imagine having the time, and so no transcript = no go.
When I’m walking to and from work, when I’m out walking on the weekends, while I’m doing chores, going shopping and so on, I do listen to a scary amount of podcasts, but then again I’m single, and live by myself, have the time to do it.
To each their own :) I did that when I was young growing up as well, going around with my walkman and diskman, but I think that was more because I didn’t have the chance to, I also had a portable radio, but the radio programmes I liked wasn’t always on air :)
They mostly replace music while I’m driving or otherwise traveling. Or while doing chores around the house; I’ve got a set of isolating (not noise cancelling) earbuds that I wear when mowing the lawn for ear protection anyway, may as well learn something while otherwise mindlessly walking up and down the yard.
That makes sense — I suppose I find my brain so taxed during my non-{moving from place to place} time that I really appreciate downtime/space for myself when I find it. (Or to put it another way, for me, mindlessly walking up and down the yard is great and much needed. At least, it was when I had a yard to mow!)
Hah, yeah, for me it’s tough to not just chew on work stuff while doing things like mowing the lawn without an explicit distraction to steer my attention elsewhere. Often though I try to listen to podcasts that aren’t directly related to software stuff to help with that. I work with UAVs so listening to e.g. military (manned) aviation podcasts is far enough away from the daily grind to help not think about it, but also has had a bunch of eureka moments where they talk about some aspect of their careers that can be applied to something I’m working on.
While doing chores, cooking, cleaning, or running. Podcasts and audiobooks are the only way I get through mindless chores. I just can’t do two language things at once, so I never quite get around to sorting the mail.
This is a wonderful piece of work if what you need is Model-View-Update on the frontend and be able to share (some) code with the backend.
What I need instead, is basically a strongly-typed LiveView ([1], [2], [3]). There’s sprocket which I’m hoping is going to fit that bill eventually. LiveView-like approaches massively simplify webapps architecture.
Maybe a bit of a lazy question - how does Eyg compile algebraic effect handlers? A cursory look at the source code suggests to me that Eyg only supports a tree-walking interpreter, and stores effect handlers to be applied as they are discovered. Is this correct?
Your cursory look is correct. There is only an interpreter and no compiler for Eyg. I have some research around what compiling them would take, but that’s all. I understand that because I support multiple resumptions compiling is harder, or at least the simple version doesn’t make so efficient output.
Gotcha. Yeah, all the implementations I am aware of require some pretty heavy lifting unless there is runtime support (translation to a monadic style, and then often a few optimization passes). I am pretty curious for “simple” compilation models in this space, and have been working on one but am in a similar spot to you. Thanks for sharing.
Based on the source this looks to be written in gleam, very cool! Any resources on actually playing around with the language locally and trying out some simple programs?
For now it always runs inside the browser. so using the version at eyg.run has the same capabilities as running that webpage locally would have. There are instructions for starting the site locally here https://github.com/CrowdHailer/eyg-lang/tree/main/website.
When will people learn? 30 years of Symonyi’s work, the countless hours put into IntelliJ’s MPS and Eclipse’s XText, half of Bracha’s thankless toiling, all ended up being ignored. Structural editing needs to operate “in the background” the way treesitter does. People are used to editing text. Let people mangle the text how they see fit, and just update your model when it parses. I’m the first in line to rail against plain text as the storage format and API of a programming language, but it is the best user interface for actually writing code when you’re trying to get ideas out of your head into the computer. It doesn’t have to be plain monospaced bollocks though, let people format it, set their own colours for bindings, insert tables, put images in comments, etc.
No one ever made progress by giving people more of what they already had.
it is the best user interface for actually writing code when you’re trying to get ideas out of your head
It is easier to learn text. And you already know text. That doesn’t mean it is the best ui. I also maintain that there haven’t been any really good structural editors yet, though fructure and hazel look interesting—real advantage comes from large-scale manipulation and analysis (incl. programmatic—take semgrep-type as one random example, but can be much more powerful and simpler; also smalltalk class browser). Text-based interfaces may converge on something in the ballpark of what you can do with pure structure eventually, but at the cost of much pointless complexity and—still—some power (eg presentation types for ast nodes), and giving back nothing in exchange but familiarity and a bit of ease of learning. If what you want is to get ideas out of your head, code may not be the most appropriate medium—that’s fine. I often work out ideas with pen and paper (NB. not text either!).
I don’t think that maintaining bidirectional mapping between text and AST is complex, it’s just little known how to do that. If you factor the cost of associated tooling which just works with text and needs to be re-implemented for each non-text language (editor, vcs, online forge), it seems that working with text is a couple of orders of magnitude less complex?
My impression is that JetBrains basically figured out how to do syntactic and semantic aware coding in early aughts, and then everyone ignored this cluster of ideas for practical usage until LSP and Tree-sitter came along.
ASTs are simple and hierarchical. If the goal is to support complex, graphical relationships, this becomes much more challenging. You would at the very least need weak pointers and in-band signalling—bad things, both.
editor, vcs, online forge
Editor I have argued is simplified by operating only on structure; unless you disagree, leave that aside. The basic function of a vcs can work just the same on opaque files (arbitrary serialisation format—necessary, but incidental and trivial); patch/diff you want to be language-aware anyway. Online forge I don’t find too interesting—in particular, its functions should be subsumed by the editing environment; e.g., you should be able to have strong pointers between tickets and code elements—but for the ‘online’ part, you may perhaps compile your editor to javascript or wasm.
More broadly, conceptually unintegrated (cf brooks) programming environments are bad; generic tooling is going to be suboptimal anyway. The preponderance of independent, disconnected programming languages and environments is an unfortunate accident that would not have been necessary given better abstractions, though it may not have been avoidable (see gabriel): the hegemony of mediocre unix meant that it was necessary to make better programming languages on top of it, but they could only interoperate at the level of its abstractions, which were limited to flat, inexpressive text, which was of necessity somewhat removed from the far richer and more useful structural abstractions used within those languages. A better-abstracted and better-constructed system would admit structural interoperation.
Note that Sophistifunk is talking about writing code while also maintaining a model which isn’t plain text. In that scenario, structured manipulation across the code base is still possible via the model alongside having localised “freeform” text entry. Personally I lean towards your view, but the freeform entry direction is also worth exploring I think.
writing code while also maintaining a model which isn’t plain text
Yes—‘projectional editing’. But it becomes increasingly cumbersome to maintain the textual mapping—hence ‘much pointless complexity’—if you want to go both ways. If it’s a one-way mapping—write a new function as text, compile it to structure and integrate it with the running system, and manipulate it thus thereafter—then fine, I suppose, but also kind of pointless imo. I’ll refer also to a brief exchange on the matter I had with geoff langdale.
Taking this further: I find it completely reasonable that a good structured editor might support complex transactions which can not easily be decomposed into structurally and semantically coherent small-steps. But that does not at all mean that textual manipulation is the best way to implement those transactions.
What I mean is that you maintain a structured model of the code that is used to drive the rest of your process, and also to derive useful information to help the human editing the code, but don’t restrict the user to operations that are atomic with respect to maintaining a correct model. You can allow the user all the wonderful model-aware operations should they chose to trigger one, but don’t force them to do so. If they just want to mess with the text let them do it, and simply maintain your last-known-good model of the text while they do so. When the text once again becomes parseable, then you update your model and re-run any dependent inspections / start showing errors / whatever you would normally update if they had made the change wholesale as an atomic operation.
In structural editors where it’s touted that the user “can’t put the source into an invalid state” it means that the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be, which can and often does present a serious extra cognitive load on the user, as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition.
Hello I am making a multi-language structural editor.
the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be
Indeed, while text can be edited with “left, right, up, down, backspace, newline, characters”, trees have more structure and so require more basic commands. Mapping it out, having a set of commands that’s about as powerful as the basic Vim commands seems to require roughly the same number of keys, with related but tree-y functionality. Here’s a sketch for a Vim-like key map (just the first set; everything after “——” is old notes):
There’s one thing you need to know while editing that isn’t present in the key map: when inserting a node, you need to say what node type it is (e.g. “+” or “function” or “if”). Our plan is that there’s a single key for each (e.g. “+”, “f”, “i”) that is shown on screen when you go to insert, to make them discoverable. This should actually increase discoverability compared to a text editor, when editing a language you’re not very familiar with.
as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few? I’m very curious what obstacles there are.
One important thing is that — at least in the structural editor I’m making — is that there can be “holes” in the document. For example, showing a hole as ?, an if/else whose condition hasn’t been filled in would be if (?) { x += 1; } else { x -= 1; }. So there’s a little bit of allowed syntactic invalidity that might make it easier to traverse between syntactically valid states.
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few?
Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes. Every single time that’s happened, you’d have had to stop and re-think not just “what do I want the code to look like when I’m done with this edit?” but also “how do I get to where I want to be using only the allowable operations within this editor?”. And perhaps when you’re the one building the structural editor this seems like it’s not a burden, but as somebody who needs to do stuff like this all the time, I can assure you it’s not. I’m not interested in adding any barriers to my job, I want tools to help me, to work for me, not constrict me in order to make their own implementation easier.
Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes.
Can you be more specific, like with actual code examples? Because I move things around all the time, and the program tends to stay syntactically valid the whole time. Or rather, it sometimes becomes invalid but only because of silly punctuation things that a structural editor would take care of. Some examples:
Example 1: swapping two list elems in JSON by cutting and pasting.
In a structural editor, you cut the third element and paste it before the first element, then cut the first element and paste it after the second element, and at all five of these steps the commas are correct.
Example 2: wrapping some stuff in a function.
Say you realize that some code ought to be inside a function so you can re-use it, and you write half the function but don’t finish it:
fn big_function() {
...
forward_map.insert(key, val);
reverse_map.insert(val, key);
...
}
-->
// syntactically invalid because key doesn't have a type
fn helper_function(key) {
forward_map.insert(key, val);
reverse_map.insert(val, key);
}
fn big_function() {
...
}
with holes, this is actually “syntactically valid”:
I like to keep my program syntactically valid, because my auto-formatter only works when the program parses. So it’s possible that you edit differently than I do, going through more invalid states. If so, I’m very curious what states those are.
A wise man once said that critics are your best friends because, unlike your fans, they help you improve your work with constructive feedback. Won’t you be my friend?
Uhu, I also tend to think that anything you can do in a structural editor, you can do in a normal editor as well, plus you’ll get all nice pure-text editing features (e.g, multiple cursors) for free.
The only issue here is that for this to work you also need a parser which doesn’t produce a train wreck of an AST if there’s a missing semicolon, and there’s frustratingly little information on how to do that. https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html explains the trick.
All of them? Speaking for at least one person I like editing the structure directly.
Probably I should be clearer. I’m building the language for myself and (if there are any) like minded people. The project (language & editor) are optimised for exploring ground that is new and interesting to me and not for mass appeal. For potential mainstream appeal I point people at https://gleam.run/ anyway.
Thanks. I actually think effect systems are going to find use in the future in many languages.
I was thinking of describing eyg as a strongly typed Lua for a while. And dropping the structural editor, but for now I’m keen to keep both experiments going. even if it makes the language a bit more niche in the meantime.
Gleam looks more and more appealing with such nice libraries.
Slightly out of scope but, it would definitely help adoption if there was a simple example of deploying such an app on a platform like Heroku.
I’d be happy to help on the Heroku part, I’m very unfamiliar with the Erlang deployment process.
Hello! Deployment documentation is certainly a top priority, and something we consider a responsibility of the core team!
Right now I’m working a lot on tooling so it’s hard to write documentation as it’s in a state of flux. For now users will have to refer to the Erlang documentation I’m afraid https://www.rebar3.org/docs/releases
For now we could have a guide about using heroku + Mix/Elixir with some Gleam modules.
That’s what I am currently doing for my Gleam deployments, though my set up is rather messy.
I’ve been thinking a lot lately about how deeply underutilized our visual cortex is for programming tasks, so this is good timing.
I think there’s a lot to learn about this from data visualization experts, but the example in the article with the bar-chart vs scatter-plot drives home just how difficult it can be to take a set of data and produce useful/interesting visualizations that communicate useful findings.
One thing that popped into my mind while reading this was of alternative uses for textual highlighting. There’s all kinds of information that could be useful when reading code, and using text color could be really useful - my first inclination was a sort of code “hotness” rating that highlights functions differently based on how widespread their use in the rest of the codebase is, or the number of times a particular function has been changed (a sort of “Chesterton’s fence” flag for functions with lots of covered edge-cases.)
Thanks for writing!
I think this comment chimes with what I took away from the post. using the visual cortex more could be powerful and it we don’t have a better answer why not start from the text. I’ve been thinking about this while building EYG a language for a projectional editor. Technically any visualisation would be possible but the text version was the smoothest experience for me.
There are lot’s of different use of highlighting I have thought about.
I was listening to your podcast with Richard Feldman, and I found it more insightful than I originally was anticipating. I forget exactly what though. I’ll have to go back and listen.
With the first two, it sounds like you kind of want a preview of how something is tied to all other parts of the code base. And because code relations are multidimensional, we don’t visualize it all at once. So perhaps like how smalltalk’s browser looks (emulated by apple’s column view), but much smaller, like a sparkline. Then you can tell how many other classes or methods in the system are affected and how far away they are from the current line of code.
I think I’d want some situational awareness of how changing one thing affects anything else in the codebase. I would think either sourcegraph or language servers would have more of this information in it, and might be available to provide this info in addition to git. It’s just that we haven’t hooked it up per a given use case while reading or writing code.
visual studio kinda does that: it inserts a “n references” line above each function, at least in C#. you can click it to see the references
it’s not a heatmap, so it takes up space in the editor, though. i think i’d want distinct colours for 0 and 1, and then a gradient for 2+. knowing zero/one/many is useful for refactoring, so distinguishing between them would be nice
I think you can apply code “hotness” for a variety of different definitions of hotness. It could be “last places where things were edited” or “places where there was a security vulnerability”. Or even a kind of “reverse observability”, where information from the production server is fed back into your IDE, so you can see which code paths are hot in real time.
I recall one of my professors working on a tool that showed (simulated) energy consumption with syntax highlighting, for low-power embedded applications. Almost literal hotness!
Renamed to “Algebraic effects are a functional approach to manage side effects”
Thanks, but the old title is still being shown on the front page. Maybe ping @pushcx for help?
It can be changed by users if enough people suggest it.
How about “Functions don’t kill people; side effects do”?
I liked the article (title aside) but wished there were some more details about how effects work.
You alluded to effects being part of a function’s type signature. It looks as though this is invisible in the source code; the compiler sees an effect being performed and annotates the function with it. If a function calls another that has an effect (without a “handle” clause) the compiler tags it with the effect too, and so on.
Is it possible to declare a function’s effects explicitly, so the compiler can issue an error if it calls something that has an undeclared effect? Otherwise it sounds like adding an effect to a lower level function “taints” everything above it in the call tree with that effect, which could have unpleasant consequences.
How effects work is at least a few different things.
So it seems like I should try and write about the type system before getting on to handlers.
Not in EYG there is no possibility to declare any types they are only ever inferred. But in the general case effects end up in a function type signature so if you can annotate a function that annotation should include the effects.
I agree. I saw EYG has effects described in the docs, but it’s only 2 paragraaphs and I don’t quite get it.
The title of this article is, in my opinion, in poor taste. The article makes almost no mention of it, and I was expecting an article related to its title, and not an explanation of the author’s programming language’s implementation of algebraic effects (or at least a link between the two that was more than a couple of sentences.)
There also isn’t an explanation as to why it would be computers and not the program (or here, the programming language.) It’s an important and interesting discussion, but it’s used only to bait-and-switch; and in particular, by using a variant of the known sentence that removes all responsibility from the program’s authors.
For the confused (like myself until a moment ago), this article used to be called “Programs don’t kill people, computers do”.
I think you make a fair point here. I’ve been iterating on the post for a while and hadn’t paid attention to the fact the last edit removed any mention. The cliche that side effects might “launch the missiles” is referenced only in the images. There is also no point in an explainer article having slight reference to “in jokes”.
This is an interesting thought. I really only wanted to emphasize that programs only act on input and have no “real” knowledge of the outside world. I’m hadn’t been making commentary on the program authors responsibility. Though as you correctly indicate also an interesting discussion.
I will think of an alternative title. “Explaining algebraic effects” might be all that’s needed.
It would be better without the clickbait title. Maybe fewer would read it, but those who do will take it more seriously. There are genuine ethical issues involved in authoring software; let’s not cheapen them.
I think you mean cliche.
indeed I do thank you.
I listened to several talks from the “sprites” (name made up on a whim just now) at fossdem. I’m very interested in finding out more about the early days and how it came into existence. It seems a genuinely interesting set of experiments and I hope to keep more up to date with what they do.
Thanks! The rest of the sprites and I are glad you enjoyed the FOSDEM talks.
I really like how he considers things through the charts he shows (although I’d choose different elements and positions), especially the idea behind the problem space one at 2:45.
3:45’s language features chart evokes cool visualizations in the direction of https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.pdf (though his is just good vs. bad). 9:30’s version seems to have better axes, but keeping things in the same position baffles me. (I’m not sure how to interpret a dichotomy between indirection and abstraction either… indirection’s a mechanism to build abstractions or subordinate detail, no?)
His note on Gall’s law (complex working systems evolved from simple working systems):
is quite insightful! Traditions within a (language) community form around shelling points, which endure by inertia as more features are added. When a language appears with a massive primitive set or no clear (best) way to do something (Unison, Flix, people flounder around and adoption doesn’t happen. There’s a clear relation to “single paradigm” (or PEP20) languages gradually branching out with feature bloat. (On the other hand… Perl? I’m not sure how true this is.)
Calling things with delayed benefits vegetables is a great turn of phrase. I will start saying EYG.
I disagree with his idea that Moore’s law enables more expressive, predictable paradigms because Forth & APL excelled with minimal resources (even if Lisp, Prolog, Haskell or Rust’s compiler require modern processing power.) Rust may have 0 cost abstractions, but some paradigms actually shift the paradigm!
Unfortunately, I’m close minded and hate the EYG (the language)’s syntax (inelegant notation constraining thought) although he presents well thought-out semantics.
is a very interesting phrasing. I’ve always thought about enforcing invariants and protecting internal logic’s correctness but somehow never thought in the opposite direction, to protect the environment from a runaway program. When he later says “the outside world calls the continuation”, well huh, I suppose the environment does choose to start a program (besides SRE considerations like powering the computer running it etc.)
!capture’s cool. I like capture the AST and passing it to another environment! Hopefully some future OS lets us even transfer closures to other programs.Thanks for the thoughtful comment. To me indirection has negative connotations, though maybe to be clear I could say obfuscation instead. That’s a fair point about Gall’s law not really playing out for Perl, probably a similar argument applies to languages like Ruby. Syntax always gets strong reactions, I already know of one effort for someone to write their own syntax on top of the EYG AST. Capture has been a really fun discovery I really hope to make a lot of it in some upcoming projects.
Is a license for EYG specified anywhere?
it’s currently unlicensed. I think I want to separate the language from the editor from a few other things in the repo and license them differently.
I’d also love to extend an invite to the BABLR community. We have a ton of alignment with your mission: we’re trying to take it the next step by offering an editor in the spirit of EYG’s that can work with any and every programming language.
I’ll have to drop in an say hi. I couldn’t quite work out what BABLR is from a cursory glance at your website.
Almost everyone says that. I’m sure I could do a better job of explaining, but also part of the problem is that it’s a new kind of thing – in a category of its own. I feel like it aught to be easiest to explain it to you though! It’s no more and no less than what is needed to make an EYG-ish editor experience for an arbitrary programming language, even a complicated real-world language like Javascript.
Like you I think that having a browser native experience gets code editing into a lot of exciting spaces that programming needs to be like TVs in homes and tablet computers in classrooms – even and especially places where code literacy is needed but installation of a native application would be a major barrier to access.
To build structural editing for an arbitrary language you need some kind of system of parsers to define the languages and some kind of universal node-tree representation which has the property of being able to snap nodes together as if they were lego bricks, just like the EYG editor does.
CSTML is the bottom layer of my infrastructure: the lego bricks. You could think of it as a reimagination of XML that learns from JSON and GraphQL to fix XML’s flaws. It has three big innovations in it:
<*BooleanLiteral> 'true' </>. Having inner text ensures that we are never at a loss for how to print the code the tree is meant to be printed, which is a very important property for being able to make editors. CSTML is not meant to be seen directly except by developers, rather it is meant to be a data layer which can encapsulate any state that might be produced by a powerful GUI tool.On top of CSTML is BABLR, the schema validation layer. It helps you narrow the infinite space of arbitrary valid CSTML down to just what should be valid according to the rules of your programming language. “Parse don’t validate,” is our philosophy, so once we’ve parsed nodes we make them immutable. Deeply immutable trees of syntax nodes lead to easy caching and another editor feature we want: the ability to hold history efficiently just by holding on to prior roots of the immutable data structure.
This works very well as an explanation for me. An admirable mission. The EYG editor was challenging and I was allowed to mold EYG the language to my purpose.
The definition of operating system in this article is so broad as to be meaningless. Words have meanings, and while the area of systems software on the whole is indeed quite nebulous, an operating system is a relatively specific subset that does not extend to Kubernetes or the a bytecode VM that runs as an application program.
The edges are still fuzzy of course, but I would expect at least a few things to be true generally:
Kubernetes is ultimately application software that runs on top, using OS facilities and providing complex orchestration to co-ordinate a fleet of machines.
If your software uses the Berkeley sockets API to co-ordinate with other instances of itself over a regular TCP/IP network, it’s not the operating system.
Timothy Roscoe defines an OS as
video slides
On a cloud VM, is the hypervisor the OS or is the guest OS the OS?
This is, honestly, the reason that the typical definition of “operating system” doesn’t really make sense anymore. Layers of operating systems. Will the real operating system please stand up?
The hypervisor is the OS for the hardware, Linux is then the OS for the guest, virtual machine… in the traditional sense.
Like I said, the edges are a bit fuzzy, but there have been virtualised machines with hypervisors for a very long time, and I don’t think it defrays the meaning of the term. Generally the OS software that runs in a virtual machine is capable of operating the whole hardware, it’s just been constrained to a partition of sorts. That constrained slice could just as well have been made by a hardware partitioning technology instead of a software one. I think what makes the hypervisor and virtual machine concept so legible is that the idea of the OS inside retains its meaning.
But now you have to classify Erlang as an operating system since it can run on xen.
I feel like a broken record, but, fuzzy edges?
I’m hugely sceptical of the value of unikernels, but it’s pretty clear that the library or framework you jam into an image alongside whatever application payload is an operating system. In this case it seems most correct to say that LING is a toolkit for making a complete operating system with an Erlang-based application payload included in the image, rather than that “Erlang is an operating system”.
I’m not sure “it’s pretty clear that the library for framework you jam into the image alongside whatever application payload is an operating system” is so clear. If that’s the case, then Linux running on Xen isn’t an operating system (based on your definition above), only the implementation of the Xen ABI is, which is just a small part of Linux as a whole.
The situation is more visibly clear with Erlang on Xen, which is why I brought that up specifically. Erlang has its own process isolation model, scheduler, and memory management. Typically, it uses the abstraction that the OS provides to provide these things, and on Xen, presumably Erlang’s implementation of these things is of much greater importance, in terms of how effective the underlying hardware is used, since the Xen ABI implementation is likely much less primitive than Linux itself.
But sure, all of this might be “fuzzy edges” and “fuzzy middles,” too, me thinks.
I initially reacted negatively to this based on instinct, but have changed my mind especially because of this description:
If Erlang - or to be more specific, the Erlang VM - isn’t an operating system (especially with LING attached), then what do you call the software that ran Lisp Machines?
To be clear, I’m merely arguing that the definition of an “operating system” is no longer as simple as it once was.
The Lisp Machines ran custom hardware and were designed to run Lisp efficiently. I assume the operating system was almost completely written in Lisp, minus likely, a small set of “assembly” routines to bootstrap a basic reader and compiler, to bring up a more capable system.
I forget the exact quote but it went something like “a programming language consists of all the bits they forgot to include in the operating system” so yeah, why not?
This quote just makes my point more relevant! The typical definition of “operating system” is no longer makes much sense.
what’s your take on unikernels then? specifically thinking of mirage, although I don’t have a lot of knowledge in that domain, just read about it a while ago.
This is possibly a way to define operating systems, and I think it’s one a lot of people understand.
I think I want a general term for a system that accepts code to run. orchestration tool is a contender, but that to me does feel more focused on aspects that make up a clustered system.
I guess at that point that you would consider DCOS misnamed https://dcos.io/
This is called an “OS” because “distributed orchestration system” isn’t as recognizable from a marketing perspective. We can’t allow technical definitions to be redefined for marketing purposes.
Imagine the insanity of someone selling square roots of numbers, but marketing decided to call them qwijibos instead. Absolutely ridiculous! They are square roots!
The web site is https://eyg.run. I couldn’t find anything about a source code repository, or how to install eyg on my Linux machine so that I can try it out and use it. It seems that an implementation exists, but nothing has been released yet. It looks interesting, somebody should post to Lobste.rs when there is a release, then I will spend time looking at it. Didn’t watch the video, sorry.
https://github.com/crowdhailer/eyg-lang I think
This is correct, although it probably still doesn’t have what you’re looking for regarding install instructions. The editor and running that exist on the web page are THE only up to date editor and latest runner. Working in the browser is the environment it’s for.
Not strictly related to the submission, but: when do people actually listen to podcasts? I can’t imagine having the time, and so no transcript = no go.
I’ve started walking with podcasts. it’s a good middle ground for exercise but still being productive. if that’s important.
I mostly manage it on public transport. I also find it hard (to find the time) otherwise.
Yes, that’s the only thing I can imagine doing it for me, too, but haven’t had a regular commute (PT or otherwise) in so long.
When I’m walking to and from work, when I’m out walking on the weekends, while I’m doing chores, going shopping and so on, I do listen to a scary amount of podcasts, but then again I’m single, and live by myself, have the time to do it.
That makes sense! Thanks for sharing.
To be honest, I usually just listen to music instead in that situation.
To each their own :) I did that when I was young growing up as well, going around with my walkman and diskman, but I think that was more because I didn’t have the chance to, I also had a portable radio, but the radio programmes I liked wasn’t always on air :)
They mostly replace music while I’m driving or otherwise traveling. Or while doing chores around the house; I’ve got a set of isolating (not noise cancelling) earbuds that I wear when mowing the lawn for ear protection anyway, may as well learn something while otherwise mindlessly walking up and down the yard.
That makes sense — I suppose I find my brain so taxed during my non-{moving from place to place} time that I really appreciate downtime/space for myself when I find it. (Or to put it another way, for me, mindlessly walking up and down the yard is great and much needed. At least, it was when I had a yard to mow!)
Hah, yeah, for me it’s tough to not just chew on work stuff while doing things like mowing the lawn without an explicit distraction to steer my attention elsewhere. Often though I try to listen to podcasts that aren’t directly related to software stuff to help with that. I work with UAVs so listening to e.g. military (manned) aviation podcasts is far enough away from the daily grind to help not think about it, but also has had a bunch of eureka moments where they talk about some aspect of their careers that can be applied to something I’m working on.
Often while driving or working out in the gym
While doing chores, cooking, cleaning, or running. Podcasts and audiobooks are the only way I get through mindless chores. I just can’t do two language things at once, so I never quite get around to sorting the mail.
This is a wonderful piece of work if what you need is Model-View-Update on the frontend and be able to share (some) code with the backend.
What I need instead, is basically a strongly-typed LiveView ([1], [2], [3]). There’s sprocket which I’m hoping is going to fit that bill eventually. LiveView-like approaches massively simplify webapps architecture.
[1] What Is Phoenix LiveView? (UPDATED)
[2] Getting Started with Phoenix LiveView (UPDATED)
[3] Phoenix LiveView Lifecycle (UPDATED)
I found https://hexdocs.pm/lustre/lustre/server_component.html which looks more like what I need.
Yeah I think server components could be what your looking for. Hopefully it turns out to be useful to you
excited for local dependencies. Should really make a difference when developing libraries.
yeah, I’m excited to no longer need to copy and paste versions of nakai everywhere as I work on it, hehe
Cool. Which programming languages and projects were the inspiration for EYG?
I’ve looked at a lot of languages now, and they’ve all influenced it a little bit.
Maybe a bit of a lazy question - how does Eyg compile algebraic effect handlers? A cursory look at the source code suggests to me that Eyg only supports a tree-walking interpreter, and stores effect handlers to be applied as they are discovered. Is this correct?
Your cursory look is correct. There is only an interpreter and no compiler for Eyg. I have some research around what compiling them would take, but that’s all. I understand that because I support multiple resumptions compiling is harder, or at least the simple version doesn’t make so efficient output.
Gotcha. Yeah, all the implementations I am aware of require some pretty heavy lifting unless there is runtime support (translation to a monadic style, and then often a few optimization passes). I am pretty curious for “simple” compilation models in this space, and have been working on one but am in a similar spot to you. Thanks for sharing.
Awesome. I’m really enjoying using Gleam. Also my post on Eyg being featured on the same day is cool because it’s all built with Gleam.
Based on the source this looks to be written in gleam, very cool! Any resources on actually playing around with the language locally and trying out some simple programs?
For now it always runs inside the browser. so using the version at eyg.run has the same capabilities as running that webpage locally would have. There are instructions for starting the site locally here https://github.com/CrowdHailer/eyg-lang/tree/main/website.
It is possible to run the cli locally using
gleam run cliwhich will hit this part of the program https://github.com/CrowdHailer/eyg-lang/blob/main/eyg/src/eyg.gleam#L11-L13 But that is not documented. probably what I need to work on soon but for now I’m focusing on making the editor good.When will people learn? 30 years of Symonyi’s work, the countless hours put into IntelliJ’s MPS and Eclipse’s XText, half of Bracha’s thankless toiling, all ended up being ignored. Structural editing needs to operate “in the background” the way treesitter does. People are used to editing text. Let people mangle the text how they see fit, and just update your model when it parses. I’m the first in line to rail against plain text as the storage format and API of a programming language, but it is the best user interface for actually writing code when you’re trying to get ideas out of your head into the computer. It doesn’t have to be plain monospaced bollocks though, let people format it, set their own colours for bindings, insert tables, put images in comments, etc.
No one ever made progress by giving people more of what they already had.
It is easier to learn text. And you already know text. That doesn’t mean it is the best ui. I also maintain that there haven’t been any really good structural editors yet, though fructure and hazel look interesting—real advantage comes from large-scale manipulation and analysis (incl. programmatic—take semgrep-type as one random example, but can be much more powerful and simpler; also smalltalk class browser). Text-based interfaces may converge on something in the ballpark of what you can do with pure structure eventually, but at the cost of much pointless complexity and—still—some power (eg presentation types for ast nodes), and giving back nothing in exchange but familiarity and a bit of ease of learning. If what you want is to get ideas out of your head, code may not be the most appropriate medium—that’s fine. I often work out ideas with pen and paper (NB. not text either!).
I don’t think that maintaining bidirectional mapping between text and AST is complex, it’s just little known how to do that. If you factor the cost of associated tooling which just works with text and needs to be re-implemented for each non-text language (editor, vcs, online forge), it seems that working with text is a couple of orders of magnitude less complex?
My impression is that JetBrains basically figured out how to do syntactic and semantic aware coding in early aughts, and then everyone ignored this cluster of ideas for practical usage until LSP and Tree-sitter came along.
ASTs are simple and hierarchical. If the goal is to support complex, graphical relationships, this becomes much more challenging. You would at the very least need weak pointers and in-band signalling—bad things, both.
Editor I have argued is simplified by operating only on structure; unless you disagree, leave that aside. The basic function of a vcs can work just the same on opaque files (arbitrary serialisation format—necessary, but incidental and trivial); patch/diff you want to be language-aware anyway. Online forge I don’t find too interesting—in particular, its functions should be subsumed by the editing environment; e.g., you should be able to have strong pointers between tickets and code elements—but for the ‘online’ part, you may perhaps compile your editor to javascript or wasm.
More broadly, conceptually unintegrated (cf brooks) programming environments are bad; generic tooling is going to be suboptimal anyway. The preponderance of independent, disconnected programming languages and environments is an unfortunate accident that would not have been necessary given better abstractions, though it may not have been avoidable (see gabriel): the hegemony of mediocre unix meant that it was necessary to make better programming languages on top of it, but they could only interoperate at the level of its abstractions, which were limited to flat, inexpressive text, which was of necessity somewhat removed from the far richer and more useful structural abstractions used within those languages. A better-abstracted and better-constructed system would admit structural interoperation.
Note that Sophistifunk is talking about writing code while also maintaining a model which isn’t plain text. In that scenario, structured manipulation across the code base is still possible via the model alongside having localised “freeform” text entry. Personally I lean towards your view, but the freeform entry direction is also worth exploring I think.
Yes—‘projectional editing’. But it becomes increasingly cumbersome to maintain the textual mapping—hence ‘much pointless complexity’—if you want to go both ways. If it’s a one-way mapping—write a new function as text, compile it to structure and integrate it with the running system, and manipulate it thus thereafter—then fine, I suppose, but also kind of pointless imo. I’ll refer also to a brief exchange on the matter I had with geoff langdale.
Taking this further: I find it completely reasonable that a good structured editor might support complex transactions which can not easily be decomposed into structurally and semantically coherent small-steps. But that does not at all mean that textual manipulation is the best way to implement those transactions.
I’d love it if I learned more about this right now :)
I’ve learned about structural editing recently, and I’d love pointers to good resources.
What do you mean by this?
What I mean is that you maintain a structured model of the code that is used to drive the rest of your process, and also to derive useful information to help the human editing the code, but don’t restrict the user to operations that are atomic with respect to maintaining a correct model. You can allow the user all the wonderful model-aware operations should they chose to trigger one, but don’t force them to do so. If they just want to mess with the text let them do it, and simply maintain your last-known-good model of the text while they do so. When the text once again becomes parseable, then you update your model and re-run any dependent inspections / start showing errors / whatever you would normally update if they had made the change wholesale as an atomic operation.
In structural editors where it’s touted that the user “can’t put the source into an invalid state” it means that the user needs to already know the shortcuts and/or the names for whatever each step would be to get the text from where it is now to what they want it to be, which can and often does present a serious extra cognitive load on the user, as well as often requiring a lot more steps to get from A to B, if the program needs to be at least syntactically valid at each point during the transition.
Hello I am making a multi-language structural editor.
Indeed, while text can be edited with “left, right, up, down, backspace, newline, characters”, trees have more structure and so require more basic commands. Mapping it out, having a set of commands that’s about as powerful as the basic Vim commands seems to require roughly the same number of keys, with related but tree-y functionality. Here’s a sketch for a Vim-like key map (just the first set; everything after “——” is old notes):
https://github.com/justinpombrio/synless/blob/master/doc/commands.txt
There’s one thing you need to know while editing that isn’t present in the key map: when inserting a node, you need to say what node type it is (e.g. “+” or “function” or “if”). Our plan is that there’s a single key for each (e.g. “+”, “f”, “i”) that is shown on screen when you go to insert, to make them discoverable. This should actually increase discoverability compared to a text editor, when editing a language you’re not very familiar with.
I’ve heard this sentiment a lot, but haven’t heard of good examples. Can you think of a few? I’m very curious what obstacles there are.
One important thing is that — at least in the structural editor I’m making — is that there can be “holes” in the document. For example, showing a hole as
?, an if/else whose condition hasn’t been filled in would beif (?) { x += 1; } else { x -= 1; }. So there’s a little bit of allowed syntactic invalidity that might make it easier to traverse between syntactically valid states.Any time in your life you’ve been moving things around and the text has been an invalid program for a few minutes. Every single time that’s happened, you’d have had to stop and re-think not just “what do I want the code to look like when I’m done with this edit?” but also “how do I get to where I want to be using only the allowable operations within this editor?”. And perhaps when you’re the one building the structural editor this seems like it’s not a burden, but as somebody who needs to do stuff like this all the time, I can assure you it’s not. I’m not interested in adding any barriers to my job, I want tools to help me, to work for me, not constrict me in order to make their own implementation easier.
Can you be more specific, like with actual code examples? Because I move things around all the time, and the program tends to stay syntactically valid the whole time. Or rather, it sometimes becomes invalid but only because of silly punctuation things that a structural editor would take care of. Some examples:
Example 1: swapping two list elems in JSON by cutting and pasting.
In a text editor:
In a structural editor, you cut the third element and paste it before the first element, then cut the first element and paste it after the second element, and at all five of these steps the commas are correct.
Example 2: wrapping some stuff in a function.
Say you realize that some code ought to be inside a function so you can re-use it, and you write half the function but don’t finish it:
with holes, this is actually “syntactically valid”:
I like to keep my program syntactically valid, because my auto-formatter only works when the program parses. So it’s possible that you edit differently than I do, going through more invalid states. If so, I’m very curious what states those are.
A wise man once said that critics are your best friends because, unlike your fans, they help you improve your work with constructive feedback. Won’t you be my friend?
Check out hazel and associated papers!
Hazel is a great project.
Another interesting one to look at is https://tylr.fun/ very pretty
Uhu, I also tend to think that anything you can do in a structural editor, you can do in a normal editor as well, plus you’ll get all nice pure-text editing features (e.g, multiple cursors) for free.
The only issue here is that for this to work you also need a parser which doesn’t produce a train wreck of an AST if there’s a missing semicolon, and there’s frustratingly little information on how to do that. https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html explains the trick.
All of them? Speaking for at least one person I like editing the structure directly.
Probably I should be clearer. I’m building the language for myself and (if there are any) like minded people. The project (language & editor) are optimised for exploring ground that is new and interesting to me and not for mass appeal. For potential mainstream appeal I point people at https://gleam.run/ anyway.
This point is made slightly clearer in the project README. https://github.com/CrowdHailer/eyg-lang/
Project homepage is here https://eyg.run/
@crowdhailer
I like the portability and effect system. Sending code over the wire made me think of Telescript. It’s also a nice-looking, summary page.
Thanks. I actually think effect systems are going to find use in the future in many languages. I was thinking of describing eyg as a strongly typed Lua for a while. And dropping the structural editor, but for now I’m keen to keep both experiments going. even if it makes the language a bit more niche in the meantime.
Gleam looks more and more appealing with such nice libraries.
Slightly out of scope but, it would definitely help adoption if there was a simple example of deploying such an app on a platform like Heroku. I’d be happy to help on the Heroku part, I’m very unfamiliar with the Erlang deployment process.
Hello! Deployment documentation is certainly a top priority, and something we consider a responsibility of the core team!
Right now I’m working a lot on tooling so it’s hard to write documentation as it’s in a state of flux. For now users will have to refer to the Erlang documentation I’m afraid https://www.rebar3.org/docs/releases
For now we could have a guide about using heroku + Mix/Elixir with some Gleam modules. That’s what I am currently doing for my Gleam deployments, though my set up is rather messy.