two interesting visual languages, orca and werkkzeug. orca is more of a cellular automaton, while werkkzeug is more of a “nodes are functions” style, but both omit the edges and use vertical adjacency to pass data between nodes
in werkkzeug, functions are stacked vertically. if a function takes multiple arguments, you stretch it wider, and put multiple boxes above it. it has save/load-variable functions to allow non-local argument passing, and re-using values in multiple places
in orca, most operators look for inputs to the left and right, and produce an output below. this ends up giving the same sort of thing, where pipelines are stacked up vertically. it also has save/load operators, and “jumper” operators to do non-local things
both of these tools are used by people to create art. so even if orca looks like befunge, it’s got a lot more thought put into making it usable. i think they’re onto something with the implicit data flow
cool. i haven’t really followed the tools inspired by werkkzeug (i assume it was the first? if you know of any that came before it, lmk), but assumed they existed
my friend wants me to get into modular synths, so maybe i should try some of these in order to save money
Context: author of a node-based environment for artists (https://ossia.io)
I think node-and-wires is popular because visual programming designers make the fundamental assumption that the underlying nature and logic of programming is just traditional textual programming.
To me, it’s actually popular because it mimicks the physical tools that people in my field are used to - there’s a lot of plugging real-life cables in and out of real-life boxes involved. I tried all my PhD to not go towards node and wire but in the end that’s the only tool that existing practitioners are familiar enough to grasp quickly.
Much of the power in pure functional programming lies in the power of higher-order functions, and I haven’t seen very good node-and-wires representation of that.
I’d recommend trying OpenMusic
To me, that’s damning evidence against the practice of using nodes-and-wires to model functions. Text is still the better form for expressing the underlying logic of functional programming.
I think that’s just because of education and habit. If you taught people visual programming that’s what they do - there are fields (computer music, game design especially with unreal engine) where the work is 99% through visual programming tools and that’s because that’s what the universities teach - and the result is that the workforceonly knows to use visual tools.
Imperative programming with node-and-wires fares no better. A loop in LabVIEW gives no more advantage or clarity over writing it in text
You don’t know how many people would do absolutely anything to not have to type or read text. I’ve seen people do stuff like hundreds of duplicated nodes in blueprints - when shown a dozen lines of code doing the same behaviour the response would be “but with blueprints I can understand it”
Cool, I looked at the PDF. Every function node has a “lambda mode”, where if you trigger it, the output is the function itself, rather than its output. https://hal.science/hal-00683472/document Thanks.
when shown a dozen lines of code doing the same behaviour the response would be “but with blueprints I can understand it”
Do you know if they understand it because they developed idioms? So that certain node groups are arranged in a certain way to be recognized by their visual pattern of nodes at a zoomed out level? Kinda like how APL has idioms? Is that why it’s more understandable to them, or is it something else?
Do you know if they understand it because they developed idioms?
the main thing I’ve seen is that people will go to absolutely any length to not have to type on their keyboard and do stuff only with the mouse. I had a boss who would purposely unplug their keyboard when testing the software we were developing to make sure everything would work without the mouse. I’ve given a talk once in a computer music / media arts conference and some people stood up and left the room when I showed some example of how to do something more easily (to me) in code with javascript than graphically.
That said, having visual patterns through layouts of nodes in graphical languages is extremely common - you see students starting to do it after a couple hours of using such a software usually.
Hmm… they’re both dataflow models, but at different levels of abstraction. RALA is superficially similar to cellular automata, except it’s async and spatially heterogeneous. It’s sort of like a hardware description language but with some unusual constraints. The propagator model is not so spatially constrained, and while it’s mostly presented in the paper using Scheme code, I always thought it was a good candidate for visual programming. You could maybe think of it as a little like pd but with special data cells holding and merging values, which gives constraint solving and bidirectional computation from simple primitives. Both are inherently parallel.
Dataflow and concatenative languages are sort of inherently spatial (and so at least potentially visual), which gives an interesting form/function alignment. One I learned about recently, which includes many concepts (and much of the overall design) from APL, is Uiua.
The propagator model is not so spatially constrained, and while it’s mostly presented in the paper using Scheme code, I always thought it was a good candidate for visual programming.
You’re right, I think - here’s a visual programming environment for propagators: https://www.holograph.so/
Though node-and-wire propagator programming does seem like it can get messy if you need to connect too many cells. I’ve had the thought recently that a graphical environment more like a schematic diagram (eliding some connections further out from each cell) might be well-suited to propagators.
Bringing it full-circle in a way, since propagators were developed in part to model electrical circuits :)
This topic always reminds me of the time when video games were changing from 2D to 3D. Studios knew how to design games, but the controls suddenly didn’t work anymore. A couple key things stick out to me that made it work:
The analog stick. Change the human interface device used to play the game
Perspective. Intelligent camera placement by default (never behind a wall, or otherwise hiding things you need to see), with manual overrides (enemy lock-on, centering)
The rest from there was universal game design, but figuring out the control scheme had to come first. Even with the right human interface device, it took the software a few cycles to catch on (Mario 64, OoT nailed it)
I’m not saying that the future of visual programming is analog sticks, triggers, and buttons. I’m saying that we’re stuck thinking in metaphorical (and literal) 2D, and are waiting for the metaphorical (literal?!) 3D to happen for building software. It won’t be a first-person camera walking around a maze pressing E to invoke functions, in the same way it won’t be node-and-wires.
I think I’m just drawn to the comparison in spirit, not so much because I think it has hidden wisdom that will make visual programming work.
Check out BABLR! CSTML is our algebra for notating code. Your article really resonated in terms of the struggle I’ve had even trying to understand what the product was that I was building. There is no existing category for this product. The first part of it was realizing that visual programming shouldn’t be language-dependent, i.e. to start programming visually I shouldn’t first have to change to writing all my programs in some new, strange visual programming language. I intend to do visual programming, but the languages I’ll make it possible to edit visually will be… all of them.
Once you reach that conclusion some things about the form become more clear:
-You need extensible parsers. In order to cover all languages the cost of making a new language that is closely related to an existing one must be small and fully incremental
You need a universal syntax tree node to make the abstraction work. A visual editor is like a set of lego bricks, and a syntax node is a lego brick. You need some sort of very well known shape for these things so that assembling a tree from nodes is just like snapping a bunch of lego bricks together.
You need “gaps” which is to say places where a syntax node is known to be missing. These are like the negative spaces in the lego bricks which make it for other bricks to be snugly connected.
You really want a system that allows you to rewrite syntax on the fly so that comprehension can be aided by on-the-fly reformatting done to the specifications of the code’s reader instead of its writer. If I write the code and I like double quote strings syntax-highlighted green and you like single quote strings highlighted red, we should use your preferences at rendering time since the goal is to aid you.
Once you assemble all these requirements there’s only once kind of existing software I know of that has requirements similar to this, and that is a web browser. That’s why i would say that the BABLR mission is to create a universal Document Object Model for code.
Existing editor projects like Hedy, Hazel and
and Pantograph prove to me that I’m not the only person interested in the expressive power of gaps. My work-in-progress editor (https://paned.it) is the first of these tools that takes this idea to what I think is its logical conclusion: being able to just dragg syntax nodes around like they really are lego bricks.
I wasn’t sure how to type things into the editor, so all I could see that there was some hover over highlighting. I think I got it once you mentioned Hazel. I’d been following that work for a little bit. I assume you’re talking about structured editing, where you get information about typed holes (gaps) in your syntax? A whole other popular type of visual programming I hadn’t covered were the block-based approaches, where Scratch is a poster-child.
I think this approach is neat, and I do wonder why structured editing isn’t the norm, but I never dove deeply enough to figure out a reason why. The visual aspect helps with the structure of the code as it relates to syntax, as I understand it. While I think there’s potential in this direction and is often under-explored, it doesn’t quite fit the “leveraging visual cortex” part. I might have to write a follow up.
Look forward to your continued progress with BABLR!
You can double tap a particular symbol to enter edit mode and be able to type on it. If a particular node is selected (not in edit mode, mind you), then you can click-drag to pick it up and set it down into an available gap. Still a very limited/incomplete feature set, but the actual UI code here is just being born – it will improve rapidly.
Compared to something like Scratch, BABLR engages your visual cortex much more thoroughly because while it conceptually maintains bricks, the primary data presentation that you interact with when reading and writing is still syntax. There is simply not any more data-dense way of communicating algorithms than the syntax of a high-level programming language.
As to why structured editing isn’t the norm, it kind of is. I’m guessing you write code with a brace pair matcher right? E.g. inserts {} when you type {? That’s a structure editing feature right there. The problems that prevent it becoming a more complete paradigm are all in the data model: the data model VSCode uses lacks any means to represent edit states which include structural holes, and changing that would require changing the data model and thus rewriting the entire ecosystem.
All the other major tools/IDEs are in the same bind, but they’re so invested in the punchard status quo of text file programs with AST structures attached to line/col offsets that there’s no longer any incentive at all to innovate. All the users have accepted that the tech is the best it will ever be, and there’s just too much moat in the plugins ecosystem for anyone (sane) to think about trying to break into this market (other than just by forking VSCode)
I’ve been thinking a lot lately about how deeply underutilized our visual cortex is for programming tasks, so this is good timing.
I think there’s a lot to learn about this from data visualization experts, but the example in the article with the bar-chart vs scatter-plot drives home just how difficult it can be to take a set of data and produce useful/interesting visualizations that communicate useful findings.
One thing that popped into my mind while reading this was of alternative uses for textual highlighting. There’s all kinds of information that could be useful when reading code, and using text color could be really useful - my first inclination was a sort of code “hotness” rating that highlights functions differently based on how widespread their use in the rest of the codebase is, or the number of times a particular function has been changed (a sort of “Chesterton’s fence” flag for functions with lots of covered edge-cases.)
One thing that popped into my mind while reading this was of alternative uses for textual highlighting.
I think this comment chimes with what I took away from the post. using the visual cortex more could be powerful and it we don’t have a better answer why not start from the text. I’ve been thinking about this while building EYG a language for a projectional editor. Technically any visualisation would be possible but the text version was the smoothest experience for me.
There are lot’s of different use of highlighting I have thought about.
One is to hover over an expression in the code and have it highlight that is needed to evaluate that value. or visa versa all the code that depends on that expression being evaluated.
Effects, I have an effect system and so all function calls can be highlighted by their effect. I had different colors for each effect and a dark mode node when the function was polymorphic in it’s effects.
More can be done with git integration. A heat map on how often the function changes would be interesting.
I was listening to your podcast with Richard Feldman, and I found it more insightful than I originally was anticipating. I forget exactly what though. I’ll have to go back and listen.
With the first two, it sounds like you kind of want a preview of how something is tied to all other parts of the code base. And because code relations are multidimensional, we don’t visualize it all at once. So perhaps like how smalltalk’s browser looks (emulated by apple’s column view), but much smaller, like a sparkline. Then you can tell how many other classes or methods in the system are affected and how far away they are from the current line of code.
I think I’d want some situational awareness of how changing one thing affects anything else in the codebase. I would think either sourcegraph or language servers would have more of this information in it, and might be available to provide this info in addition to git. It’s just that we haven’t hooked it up per a given use case while reading or writing code.
visual studio kinda does that: it inserts a “n references” line above each function, at least in C#. you can click it to see the references
it’s not a heatmap, so it takes up space in the editor, though. i think i’d want distinct colours for 0 and 1, and then a gradient for 2+. knowing zero/one/many is useful for refactoring, so distinguishing between them would be nice
I think you can apply code “hotness” for a variety of different definitions of hotness. It could be “last places where things were edited” or “places where there was a security vulnerability”. Or even a kind of “reverse observability”, where information from the production server is fed back into your IDE, so you can see which code paths are hot in real time.
I recall one of my professors working on a tool that showed (simulated) energy consumption with syntax highlighting, for low-power embedded applications. Almost literal hotness!
May not be completely relevant to the article, but I always wanted to write a blog post titled something like “Visual programming is here, right before our eyes”.
I really enjoyed learning about CellPond, but I don’t believe that we have to go to such “depths” for visual programming - a featureful IDE with a well-supported language is pretty much already that.
Programs in most PLs are not 1D text, they are 2D text with some rudimentary visual structuring, like blocks, indentation, alignment etc.
This is turbo-charged by a good IDE that makes operations operate on symbolic structures and not on a list of characters. E.g. something as simple as putting the caret on a variable will immediately “bring up another layer of information” as a graph showcasing the declaration and uses of it. I can also select an expression (and can expand it syntactically) and choose to store that expression as a new node. Intellij for example will even ask if I want to replace the same expression at different places, effectively merging nodes. And I would argue that experienced developers look at code just like that - we are making use of our visual cortex, we are pattern matching on blocks/expressions, etc. We are not reading like you would read a book, not at all.
I can also see all the usages of a function right away, click on its definition, its overrides, parent class/interface right from my IDE. These are semantically relevant operations on specific types of nodes, not text.
Also, writing is just a very efficient and accurate visual encoding of arbitrary ideas, so it only makes sense to make use of it where a more specific encoding can’t be applied (sort of a fallback). “Nodes” (blocks of code) being text-based doesn’t take away from “visual programming” in my book. I would actually prefer to *extend” ordinary programming languages with non-textual blocks, so that I can replace, say, a textual state machine with a literal, interactive FSM model, and my IDE will let me play with it. (If we look at it from far away, Intellij again has a feature that lets you test regular expressions by simply writing a sample text, and it showcases which part matches and which doesn’t). Because a graph of a state machine is more “efficient”/“human-friendly” than a textual representation. But I think it’s faulty to think that we could have a representation that will apply equally well for different areas - this is exactly what human writing is.
I care a lot if it takes 10 seconds to start! I quit vim a lot while I’m working. In part because it takes 0 seconds to start. Though the Haskell language server kinda ruins that.
Great post, and thanks for introducing me to Cellpond.
Much of the power in pure functional programming lies in the power of higher-order functions, and I haven’t seen very good node-and-wires representation of that.
This seems like a trivial problem? A node is a function. Input wires specify arguments. If one or more arguments are left open (no wire), then the function call is a partial application, and denotes a function. If all of the inputs are open, then the node denotes the function itself. Now it is trivial to pass a function as an argument to another function.
Also, here’s Tangible Functional Programming, aka Eros, a simple and powerful visual syntax for live programming of pure functional programs, that doesn’t use nodes and wires. And the accompanying video. In this design, I feel that form really does follow function.
Ah, I hadn’t come across that model for first class functions before. What do you wire up as the function? Is it the output nub, since not all input parameters are filled? Will make a note on the post this weekend, thanks.
I’ve seen Conal’s talk before! From what I remember about it–while I agree that GUIs, as they’re built today are the compositional dead-end of any computer system, this doesn’t fit the bill because it’s not using our visual cortex to reason about program state or program flow. I might have to write a follow up.
Right, the output nub of a function produces a function if not all the input parameters are filled.
I haven’t seen this model in a node-and-wire language, but I have little experience with those languages. I’ve seen it in text based functional languages. The K language works almost exactly like this, when you use the explicit function call syntax with square brackets and that syntax’s ability to create partial applications when arguments are omitted. Haskell is not far off (with its curried functions and operator sections).
Conal’s Eros language qualifies as form follows function because the basic operations of the virtual machine are mirrored directly in the GUI, and it is highly composable. Vertical composition is function-call, horizontal composition is pair formation.
Cellpond has the property that all state is visible on the screen, which is probably pretty rare. Scott Kim created such an environment, called Viewpoint, for his 1988 PhD thesis. It is very mind bending to see a textual GUI where all the state in the entire system is visible on the screen, and is fully editable. It is like nothing else you’ve seen. Scott Kim is a strong visual thinker, and he wanted to create a UI for visual thinkers.
https://www.youtube.com/watch?v=9G0r7jL3xl8
This essay (and the CellPond SPLASH talk) tickled my brain enough that I dug through some old bookmarks and found a couple more spatial programming projects about engineering emergent behavior:
Oh, swarm chemistry reminds me of Netlogo on first glance. Simulations seem like a good bread and butter.
One thing that I never quite figured out with these kinds of simulations is that the output is visual, but the program is still text. In order to have the visual program and the visual output existing in the same space, the visual glyphs has to do double duty–the same glyph has to be both the visual element of the output, and the means by which you program. A kind of isomorphism between a program and its visual representation.
Robin Milner, who you may know from the π-calculus and The Definition of Standard ML, devoted the last years of his career to an inherently distributed graphical model of computation called bigraphs. I would summarize it as a system of pattern matching and reduction rules applied to certain kinds of mathematical graphs. It’s very amenable to visualization, since it has the isomorphism property you mention.
I find the bigraph model fascinatingly beautiful and would probably devote myself to its study and development, if I were a rich man. At one time I hoped to do graduate study on the topic, but that ship has long since sailed.
I was on Firefox on iOS and I got a page with a square on it from which neither the back button nor selecting an earlier page from the history list worked to get me off. I have to close the tab.
two interesting visual languages, orca and werkkzeug. orca is more of a cellular automaton, while werkkzeug is more of a “nodes are functions” style, but both omit the edges and use vertical adjacency to pass data between nodes
in werkkzeug, functions are stacked vertically. if a function takes multiple arguments, you stretch it wider, and put multiple boxes above it. it has save/load-variable functions to allow non-local argument passing, and re-using values in multiple places
in orca, most operators look for inputs to the left and right, and produce an output below. this ends up giving the same sort of thing, where pipelines are stacked up vertically. it also has save/load operators, and “jumper” operators to do non-local things
both of these tools are used by people to create art. so even if orca looks like befunge, it’s got a lot more thought put into making it usable. i think they’re onto something with the implicit data flow
You can also combine werkkzeug-like nodes with regular noodles like in Still’s Tooll2. The picture also stays relatively clean if you visualize the save/load connections in the graph like in Brain Control’s Enigma Studio 3.
cool. i haven’t really followed the tools inspired by werkkzeug (i assume it was the first? if you know of any that came before it, lmk), but assumed they existed
my friend wants me to get into modular synths, so maybe i should try some of these in order to save money
Cardinal is a Eurorack-inspired modular synthesis system that is open source and contains hundreds of eurorack-style modules.
(im not associated with them, i just really like it)
Context: author of a node-based environment for artists (https://ossia.io)
To me, it’s actually popular because it mimicks the physical tools that people in my field are used to - there’s a lot of plugging real-life cables in and out of real-life boxes involved. I tried all my PhD to not go towards node and wire but in the end that’s the only tool that existing practitioners are familiar enough to grasp quickly.
I’d recommend trying OpenMusic
I think that’s just because of education and habit. If you taught people visual programming that’s what they do - there are fields (computer music, game design especially with unreal engine) where the work is 99% through visual programming tools and that’s because that’s what the universities teach - and the result is that the workforceonly knows to use visual tools.
You don’t know how many people would do absolutely anything to not have to type or read text. I’ve seen people do stuff like hundreds of duplicated nodes in blueprints - when shown a dozen lines of code doing the same behaviour the response would be “but with blueprints I can understand it”
Cool, I looked at the PDF. Every function node has a “lambda mode”, where if you trigger it, the output is the function itself, rather than its output. https://hal.science/hal-00683472/document Thanks.
Do you know if they understand it because they developed idioms? So that certain node groups are arranged in a certain way to be recognized by their visual pattern of nodes at a zoomed out level? Kinda like how APL has idioms? Is that why it’s more understandable to them, or is it something else?
the main thing I’ve seen is that people will go to absolutely any length to not have to type on their keyboard and do stuff only with the mouse. I had a boss who would purposely unplug their keyboard when testing the software we were developing to make sure everything would work without the mouse. I’ve given a talk once in a computer music / media arts conference and some people stood up and left the room when I showed some example of how to do something more easily (to me) in code with javascript than graphically.
That said, having visual patterns through layouts of nodes in graphical languages is extremely common - you see students starting to do it after a couple hours of using such a software usually.
CellPond looks really interesting! Thanks.
There’s this whole line of research that you might appreciate: Reconfigurable Async Logic Automata.
Less grid-oriented and hardware-centric, the Radul / Sussman propagator model may also be of interest.
Ah cool, thanks. I’ll check them out. what’s the one thing I should keep in mind or look out for when looking at them?
Hmm… they’re both dataflow models, but at different levels of abstraction. RALA is superficially similar to cellular automata, except it’s async and spatially heterogeneous. It’s sort of like a hardware description language but with some unusual constraints. The propagator model is not so spatially constrained, and while it’s mostly presented in the paper using Scheme code, I always thought it was a good candidate for visual programming. You could maybe think of it as a little like pd but with special data cells holding and merging values, which gives constraint solving and bidirectional computation from simple primitives. Both are inherently parallel.
Dataflow and concatenative languages are sort of inherently spatial (and so at least potentially visual), which gives an interesting form/function alignment. One I learned about recently, which includes many concepts (and much of the overall design) from APL, is Uiua.
You’re right, I think - here’s a visual programming environment for propagators: https://www.holograph.so/
Though node-and-wire propagator programming does seem like it can get messy if you need to connect too many cells. I’ve had the thought recently that a graphical environment more like a schematic diagram (eliding some connections further out from each cell) might be well-suited to propagators.
Bringing it full-circle in a way, since propagators were developed in part to model electrical circuits :)
This topic always reminds me of the time when video games were changing from 2D to 3D. Studios knew how to design games, but the controls suddenly didn’t work anymore. A couple key things stick out to me that made it work:
The rest from there was universal game design, but figuring out the control scheme had to come first. Even with the right human interface device, it took the software a few cycles to catch on (Mario 64, OoT nailed it)
I’m not saying that the future of visual programming is analog sticks, triggers, and buttons. I’m saying that we’re stuck thinking in metaphorical (and literal) 2D, and are waiting for the metaphorical (literal?!) 3D to happen for building software. It won’t be a first-person camera walking around a maze pressing E to invoke functions, in the same way it won’t be node-and-wires.
I think I’m just drawn to the comparison in spirit, not so much because I think it has hidden wisdom that will make visual programming work.
Check out BABLR! CSTML is our algebra for notating code. Your article really resonated in terms of the struggle I’ve had even trying to understand what the product was that I was building. There is no existing category for this product. The first part of it was realizing that visual programming shouldn’t be language-dependent, i.e. to start programming visually I shouldn’t first have to change to writing all my programs in some new, strange visual programming language. I intend to do visual programming, but the languages I’ll make it possible to edit visually will be… all of them.
Once you reach that conclusion some things about the form become more clear: -You need extensible parsers. In order to cover all languages the cost of making a new language that is closely related to an existing one must be small and fully incremental
Once you assemble all these requirements there’s only once kind of existing software I know of that has requirements similar to this, and that is a web browser. That’s why i would say that the BABLR mission is to create a universal Document Object Model for code.
Existing editor projects like Hedy, Hazel and and Pantograph prove to me that I’m not the only person interested in the expressive power of gaps. My work-in-progress editor (https://paned.it) is the first of these tools that takes this idea to what I think is its logical conclusion: being able to just dragg syntax nodes around like they really are lego bricks.
I wasn’t sure how to type things into the editor, so all I could see that there was some hover over highlighting. I think I got it once you mentioned Hazel. I’d been following that work for a little bit. I assume you’re talking about structured editing, where you get information about typed holes (gaps) in your syntax? A whole other popular type of visual programming I hadn’t covered were the block-based approaches, where Scratch is a poster-child.
I think this approach is neat, and I do wonder why structured editing isn’t the norm, but I never dove deeply enough to figure out a reason why. The visual aspect helps with the structure of the code as it relates to syntax, as I understand it. While I think there’s potential in this direction and is often under-explored, it doesn’t quite fit the “leveraging visual cortex” part. I might have to write a follow up.
Look forward to your continued progress with BABLR!
You can double tap a particular symbol to enter edit mode and be able to type on it. If a particular node is selected (not in edit mode, mind you), then you can click-drag to pick it up and set it down into an available gap. Still a very limited/incomplete feature set, but the actual UI code here is just being born – it will improve rapidly.
Compared to something like Scratch, BABLR engages your visual cortex much more thoroughly because while it conceptually maintains bricks, the primary data presentation that you interact with when reading and writing is still syntax. There is simply not any more data-dense way of communicating algorithms than the syntax of a high-level programming language.
As to why structured editing isn’t the norm, it kind of is. I’m guessing you write code with a brace pair matcher right? E.g. inserts
{}when you type{? That’s a structure editing feature right there. The problems that prevent it becoming a more complete paradigm are all in the data model: the data model VSCode uses lacks any means to represent edit states which include structural holes, and changing that would require changing the data model and thus rewriting the entire ecosystem.All the other major tools/IDEs are in the same bind, but they’re so invested in the punchard status quo of text file programs with AST structures attached to line/col offsets that there’s no longer any incentive at all to innovate. All the users have accepted that the tech is the best it will ever be, and there’s just too much moat in the plugins ecosystem for anyone (sane) to think about trying to break into this market (other than just by forking VSCode)
I’ve been thinking a lot lately about how deeply underutilized our visual cortex is for programming tasks, so this is good timing.
I think there’s a lot to learn about this from data visualization experts, but the example in the article with the bar-chart vs scatter-plot drives home just how difficult it can be to take a set of data and produce useful/interesting visualizations that communicate useful findings.
One thing that popped into my mind while reading this was of alternative uses for textual highlighting. There’s all kinds of information that could be useful when reading code, and using text color could be really useful - my first inclination was a sort of code “hotness” rating that highlights functions differently based on how widespread their use in the rest of the codebase is, or the number of times a particular function has been changed (a sort of “Chesterton’s fence” flag for functions with lots of covered edge-cases.)
Thanks for writing!
I think this comment chimes with what I took away from the post. using the visual cortex more could be powerful and it we don’t have a better answer why not start from the text. I’ve been thinking about this while building EYG a language for a projectional editor. Technically any visualisation would be possible but the text version was the smoothest experience for me.
There are lot’s of different use of highlighting I have thought about.
I was listening to your podcast with Richard Feldman, and I found it more insightful than I originally was anticipating. I forget exactly what though. I’ll have to go back and listen.
With the first two, it sounds like you kind of want a preview of how something is tied to all other parts of the code base. And because code relations are multidimensional, we don’t visualize it all at once. So perhaps like how smalltalk’s browser looks (emulated by apple’s column view), but much smaller, like a sparkline. Then you can tell how many other classes or methods in the system are affected and how far away they are from the current line of code.
I think I’d want some situational awareness of how changing one thing affects anything else in the codebase. I would think either sourcegraph or language servers would have more of this information in it, and might be available to provide this info in addition to git. It’s just that we haven’t hooked it up per a given use case while reading or writing code.
visual studio kinda does that: it inserts a “n references” line above each function, at least in C#. you can click it to see the references
it’s not a heatmap, so it takes up space in the editor, though. i think i’d want distinct colours for 0 and 1, and then a gradient for 2+. knowing zero/one/many is useful for refactoring, so distinguishing between them would be nice
I think you can apply code “hotness” for a variety of different definitions of hotness. It could be “last places where things were edited” or “places where there was a security vulnerability”. Or even a kind of “reverse observability”, where information from the production server is fed back into your IDE, so you can see which code paths are hot in real time.
I recall one of my professors working on a tool that showed (simulated) energy consumption with syntax highlighting, for low-power embedded applications. Almost literal hotness!
May not be completely relevant to the article, but I always wanted to write a blog post titled something like “Visual programming is here, right before our eyes”.
I really enjoyed learning about CellPond, but I don’t believe that we have to go to such “depths” for visual programming - a featureful IDE with a well-supported language is pretty much already that.
Programs in most PLs are not 1D text, they are 2D text with some rudimentary visual structuring, like blocks, indentation, alignment etc.
This is turbo-charged by a good IDE that makes operations operate on symbolic structures and not on a list of characters. E.g. something as simple as putting the caret on a variable will immediately “bring up another layer of information” as a graph showcasing the declaration and uses of it. I can also select an expression (and can expand it syntactically) and choose to store that expression as a new node. Intellij for example will even ask if I want to replace the same expression at different places, effectively merging nodes. And I would argue that experienced developers look at code just like that - we are making use of our visual cortex, we are pattern matching on blocks/expressions, etc. We are not reading like you would read a book, not at all.
I can also see all the usages of a function right away, click on its definition, its overrides, parent class/interface right from my IDE. These are semantically relevant operations on specific types of nodes, not text.
Also, writing is just a very efficient and accurate visual encoding of arbitrary ideas, so it only makes sense to make use of it where a more specific encoding can’t be applied (sort of a fallback). “Nodes” (blocks of code) being text-based doesn’t take away from “visual programming” in my book. I would actually prefer to *extend” ordinary programming languages with non-textual blocks, so that I can replace, say, a textual state machine with a literal, interactive FSM model, and my IDE will let me play with it. (If we look at it from far away, Intellij again has a feature that lets you test regular expressions by simply writing a sample text, and it showcases which part matches and which doesn’t). Because a graph of a state machine is more “efficient”/“human-friendly” than a textual representation. But I think it’s faulty to think that we could have a representation that will apply equally well for different areas - this is exactly what human writing is.
[Comment removed by author]
I care a lot if it takes 10 seconds to start! I quit vim a lot while I’m working. In part because it takes 0 seconds to start. Though the Haskell language server kinda ruins that.
[Comment removed by author]
Great post, and thanks for introducing me to Cellpond.
This seems like a trivial problem? A node is a function. Input wires specify arguments. If one or more arguments are left open (no wire), then the function call is a partial application, and denotes a function. If all of the inputs are open, then the node denotes the function itself. Now it is trivial to pass a function as an argument to another function.
Also, here’s Tangible Functional Programming, aka Eros, a simple and powerful visual syntax for live programming of pure functional programs, that doesn’t use nodes and wires. And the accompanying video. In this design, I feel that form really does follow function.
Ah, I hadn’t come across that model for first class functions before. What do you wire up as the function? Is it the output nub, since not all input parameters are filled? Will make a note on the post this weekend, thanks.
I’ve seen Conal’s talk before! From what I remember about it–while I agree that GUIs, as they’re built today are the compositional dead-end of any computer system, this doesn’t fit the bill because it’s not using our visual cortex to reason about program state or program flow. I might have to write a follow up.
Right, the output nub of a function produces a function if not all the input parameters are filled.
I haven’t seen this model in a node-and-wire language, but I have little experience with those languages. I’ve seen it in text based functional languages. The K language works almost exactly like this, when you use the explicit function call syntax with square brackets and that syntax’s ability to create partial applications when arguments are omitted. Haskell is not far off (with its curried functions and operator sections).
Conal’s Eros language qualifies as form follows function because the basic operations of the virtual machine are mirrored directly in the GUI, and it is highly composable. Vertical composition is function-call, horizontal composition is pair formation.
Cellpond has the property that all state is visible on the screen, which is probably pretty rare. Scott Kim created such an environment, called Viewpoint, for his 1988 PhD thesis. It is very mind bending to see a textual GUI where all the state in the entire system is visible on the screen, and is fully editable. It is like nothing else you’ve seen. Scott Kim is a strong visual thinker, and he wanted to create a UI for visual thinkers. https://www.youtube.com/watch?v=9G0r7jL3xl8
This essay (and the CellPond SPLASH talk) tickled my brain enough that I dug through some old bookmarks and found a couple more spatial programming projects about engineering emergent behavior:
Also wanted to link MarkovJunior, referenced by Lu in the SPLASH presentation.
Oh, swarm chemistry reminds me of Netlogo on first glance. Simulations seem like a good bread and butter.
One thing that I never quite figured out with these kinds of simulations is that the output is visual, but the program is still text. In order to have the visual program and the visual output existing in the same space, the visual glyphs has to do double duty–the same glyph has to be both the visual element of the output, and the means by which you program. A kind of isomorphism between a program and its visual representation.
I’ve only seen two examples that mostly get there, ink and switch’s crosscut https://www.inkandswitch.com/crosscut/ and the game Baba is You.
Robin Milner, who you may know from the π-calculus and The Definition of Standard ML, devoted the last years of his career to an inherently distributed graphical model of computation called bigraphs. I would summarize it as a system of pattern matching and reduction rules applied to certain kinds of mathematical graphs. It’s very amenable to visualization, since it has the isomorphism property you mention.
I find the bigraph model fascinatingly beautiful and would probably devote myself to its study and development, if I were a rich man. At one time I hoped to do graduate study on the topic, but that ship has long since sailed.
There’s a book and some fairly clunky research-grade software available: e.g.BigraphER, https://bigraphs.org/
Wil, this is a fantastic essay. And you’ve posted it at the exact right time for me in my career. Thank you!
Hey ya! Long time no see. No problem, glad to help. What are you working on?
Lateral drift, my friend. Lateral drift.
I disagree with webpages that bork my back button. It feels very 90s spammy.
How did it break your back button? I tried it myself and was able to back button back here.
I was on Firefox on iOS and I got a page with a square on it from which neither the back button nor selecting an earlier page from the history list worked to get me off. I have to close the tab.
Sorry, I also use FF on iOS. No idea what happened there. I didn’t put anything like that in!