I’ve tagged this with PLT as well as design because, in the philosophy of this essay, a GUI should be a natural extension of the language underlying it and vice versa (in order to provide the kinds of facilities natural to a command-line environment in a graphical one).
vvvv is pretty much what the author is talking about. The base data structure is the list and everything is built on top of that. Components can be connected together and then grouped into new components, with some fallback on code with the more complicated parts.
I’m not familiar with this, but it looks very cool!
There are actually a lot of environments that are sort of along the same lines. A poster here has written a system like this in Lua that’s quite impressive, and as I mentioned, there’s the Smalltalk environments.
A lot of systems built along these lines are for graphics, which is fine, but I’m more interested in optimizing for the use-case of letting people who don’t identify as programmers do real work in a way that has graphical representation that makes sense to them.
A lot of people have specific, complex, idiosyncratic systems for organizing their every-day tasks that they’ve built on top of inflexible things like spreadsheets or collections of existing websites, and I’d like to cater to that kind of thing – making it possible for them to automate some of that stuff and have an easier time because they can produce a structure that’s a better fit.
Excel is actually the most successful programming environment in that regard. The builtin tracing (intermediate values stored in cells) and the spacial representation makes it approachable for people who think that they are not programmers. I think that this is amazing as those same people’s mind would often freeze if they assumed it was a programming language. It’s also deployed in countless of environment and powers a lot of business processes around the world.
But as you noticed, Excel only works for specific types of problems. So far, all the attempts to produce a generic visual programming language have been a failure in my opinion so maybe the solution is to keep the language specialized and then make many of them so all the use-cases can be covered. Create a bunch of UI DSLs.
You may already be aware of this but here’s an example about composability, especially with respect to editing parents. Its not primarily “for making UIs”.
The Alto & Squeak environments get close, and so does Tcl/Tk’s ‘wish’ interpreter, but we’re not there yet.
For each of these, which of your items are missing?
Can you post some screenshots and/or describe and discuss your current implementation, even if not final? It would help clarify some of your implicit assumptions about the problem and model. (For example, what are things you would consider a widget and not a widget.)
Every object has a visual representation.
Why only one? Or do you mean at least one?
Every visual representation corresponds to an object.
Also why only one? What about visually nested things? (Is a representation something already rendered or an abstract description?)
Edit: Also, can you post some links to some examples of UIs on TV you consider good examples of what you want to do?
For each of these, which of your items are missing?
I consider Smalltalk to have too high an initial learning overhead for this purpose (since my goal is to have the composability of graphical elements be an ‘invitation’ to programming in general for casual users).
Tk has a lower initial learning overhead but there are a lot of inconsistencies in how different widgets work (along with problems like a single-thread model), and no system has been written using Tk that does composability in the natural way smalltalk environments do. (I attempted a TCL/TK implementation many years ago. It’s briefly described here. It quickly became unmaintainable, and the widgets were not flexible enough for my liking.)
Can you […] describe and discuss your current implementation
Sure!
Io is an extremely simple homoiconic language with a prototype-based object system. It looks a lot like rebol. I chose it because an important part of making a language look understandable to non-programmers is conceptual simplicity, and Io’s pointfree style and message-passing-like behavior is a good fit for a context where a lot of the coding is glue between objects.
Because of certain features I wanted that stock Io didn’t have and certain behaviors that might be bugs (and because Io is not maintained), I decided to write my own implementation in a language with primitives that are a better match for the parts of Io I consider important for this task.
Since Io has a prototype-object system and not a class-object system, exploration through cloning a working object and modifying its guts is possible at run time. I expect this to be a normal pattern for experimentation.
I have defined the ‘widget’ object as having a position, size, and bitmap. Any widget inherits these properties. So, any widget has a visual representation, in the sense that it has a bitmap & that bitmap gets blitted to the display at render time. (Of course, it can be transparent, or whatever.) If you look at my implementation, I’ve also got some logic for visually nested things: children are positions at offset with respect to their parents, always at a higher layer, and their position affects the computed size of their parent.
The importance of widgets having a visual representation is that the expected method of modifying a widget is by selecting it in the interface and inspecting its implementation. So, ideally, every live object would be a widget whose visual representation indicates its state. (I don’t plan to make non-widget objects impossible, but it’s bad form, since they’re going to be harder to find and change.)
Unfortunately, I haven’t gotten to the point of having anything render. Stock Io’s various graphics bindings won’t build for me, & my reimplementation has a ways to go. Screenshots might be misleading, though: my goal is to have a system that’s extremely mutable in its form. Squeak’s smalltalk-based desktop is a good representation of the kind of thing I’m aiming for.
can you post some links to some examples of UIs on TV you consider good examples of what you want to do?
I’ll dig up some. Alternately, TVTropes has somelists.
Think along the lines of the way viruses are depicted in the movie Hackers – as graphical objects with apparent properties. Obviously, viruses have other reasons for not exposing their nature graphically – the whole point of them is to be hidden from & hostile to the end user – but having a graphical representation that matches the end user’s mental model of a task (as opposed to skeuomorphism, wherein it matches a programmer’s model of a device that used to accomplish a similar task, or the current norm, where the UI maps onto the structure of the programmer’s code) is desirable, even though it can’t be attained unless the end user writes it themselves. (The goal here is to minimize the artificial friction users encounter in making their computing environment a natural reflection of their mind.)
Thanks, that’s a very interesting read and I understand better now.
I have defined the ‘widget’ object as having a position, size, and bitmap.
Does this mean everything will mainly be raster (as opposed to vector). What about other visual transformations? How (un)important are they? Or maybe my question actually is whether composition will mainly be on functionality or visual appearance as well?
(I don’t plan to make non-widget objects impossible, but it’s bad form, since they’re going to be harder to find and change.)
Hmm…what about overlays for viewing and editing widgets, like corners, text cursors and bounding boxes? I guess they could all be their own separate overlay widgets.
Unfortunately, I haven’t gotten to the point of having anything render.
That might not be an issue for prototyping, except possibly for testing speed. To try it out, you might have a mock “render” loop that just prints out everything that should be drawn (and where) for that frame.
And some random questions about the implementation you linked too.
I’m a bit concerned about calculateBbox and widgetLint’s potential resource (time) consumption.
For setParent, shouldn’t the current parent, if there is one, also remove “self” as a child?
What about user input? Do they just get passed up the tree from the child? What if you want to globally change keybindings afterwards? Does the prototype-object system help with this or makes it harder?
A lot of what I’ve written here is probably too early to think about (although something like vector vs raster is pretty hard to change later on).
I definitely agree with Squeak not being inviting enough. I haven’t used Tk all that much despite having made this but did find issues with its internal model a few times.
I guess if Io is simple enough, it can even be part of the thing that’s to be rewritten/customized.
In the particular system I’m writing, I’m only supporting raster, because I am lazy. I’m adopting the model from Menuet, because rendering becomes trivial: clear buffer, blit existing bitmaps onto buffer at given positions in Z order, flip. (I have the necessary information to later optimize out totally occuluded stuff, or keep a dirty bit in order to determine what parts of the display should be updated, if performance becomes a problem, but since unlike the Applet system in Java the user code doesn’t re-draw on visibility, it actually will be less of an issue, with the downside being the occasional half-drawn widget being rendered for one frame.)
whether composition will mainly be on functionality
Composition is performed through message-passing. My goal is to provide the GUI equivalent of unix pipes.
I guess they could all be their own separate overlay widgets.
Yeah. It cleans the system up conceptually if there are no “special” classes of widgets that don’t count as widgets.
To try it out, you might have a mock “render” loop that just prints out everything that should be drawn (and where) for that frame.
The stage I’m at in development is concerned with rewriting Io so that it can actually run my program.
I’m a bit concerned about calculateBbox and widgetLint’s potential resource (time) consumption.
This is a proof of concept system & so I’m not concerned with performance at all. If major performance problems appear, I’ll look at some of the low-hanging fruit (like the Io intepreter re-parsing and re-tokenizing the entire expression over and over) first.
What about user input? Do they just get passed up the tree from the child?
Yes, from the focused child.
What if you want to globally change keybindings afterwards?
That’s not a facility I’m supporting, though generally speaking you’d only be creating key bindings for widgets that use them, so having layers of conflicting triggers would be bad form.
Does the prototype-object system help with this or makes it harder?
It depends on how you use it. The inheritance hierarchy is distinct from the widget-packing hierarchy, and both are used for message-handling (of which input handling is a part). It’s definitely possible to make a huge mess with this.
A lot of what I’ve written here is probably too early to think about
I actually made a lot of these decisions before I started writing any code, so you’re absolutely right to ask these questions. Because I expect to have between zero and one users, performance isn’t a priority, and I’m ignoring it in favor of making everything structurally & conceptually straightforward.
if Io is simple enough, it can even be part of the thing that’s to be rewritten/customized.
Io is pretty bare-bones and I’m looking to write as much as possible in it, as opposed to Tk, where complex things like the text entry widget are basically exposed from FFI. I probably won’t expose the actual parser to be rewritten, though.
I’m curious – has there been any development on this since? Would you be interested in any volunteer work? I’ve got some time to kill, as I don’t start work for a while. PM me if you’re interested.
I haven’t done any work on this for a while, no. I’d love somebody to pick up where I left off, or work on their own projects inspired by this.
I’m likely to work on it very intermittently. I have a lot of side projects, and with the current load at work, I get a chance to commit to one maybe once every few weeks. This one is complicated, since I’m writing a whole programming language interpreter for it, so it’ll probably get less love than my simpler projects.
I’ve started poking at it in https://github.com/tekknolagi/kamui (having filtered out the subfolder from your misc repository). I’m very new to Io and this idea, so we’ll see if I get anywhere at all.
One of the reasons why composable text based interfaces are so powerful is that they allow you to save your work as scripts. I can imagine a system where actions are saved as macros that could be composed together, but I don’t know if I can see it achieving the same fidelity as a text based interface. Is there value in opening up example based macros for composable interfaces? If so, could they be mixed and matched and pushed back into the widget to make more complex primitives? (E.g., like creating a new brush from two basic brushes.)
On the text side, I see the prototype-object facility for widgets providing an example-macro-like functionality. Unfortunately, outside of trivial stuff (like stealing the Alto’s method of representing a composition of two applications: the gesture of drawing an arrow between open windows), I’m not sure how to expose composability in a graphical way. I’m instead trying to make the jump to text as natural as possible.
I’m certainly no expert in GUI programming, but these guidelines seem to completely miss a major source of complexity: state management and duplication. The immediate mode GUI approach is the only one I’m aware of that minimizes state complexity, though Elm looks pretty close. My shallow understanding is that React provides some of the benefits of immediate mode, but not all of them.
Edit: for an example of how powerful immediate mode GUI programming can be, watch this handmade hero episode, where Casey implements a simple radial menu in under an hour: https://youtu.be/ftZIujU3Udw
In the system I’m proposing, the general rule of thumb is that all state should be visible.
The system will absolutely get hairy. I don’t think that matters much, so long as the hairiness is the product of the primary user & is interacted with frequently enough. In this context, I am opposed to scale.
I’ve tagged this with PLT as well as design because, in the philosophy of this essay, a GUI should be a natural extension of the language underlying it and vice versa (in order to provide the kinds of facilities natural to a command-line environment in a graphical one).
The beginning of some work along these lines is here: https://github.com/enkiv2/misc/tree/master/kamui
I’ve written about some of these ideas elsewhere:
vvvv is pretty much what the author is talking about. The base data structure is the list and everything is built on top of that. Components can be connected together and then grouped into new components, with some fallback on code with the more complicated parts.
see https://vvvv.org/screenshots
The main caveat is that it only works great if your data is lists or stream-oriented.
The language is not very well knows but also actively used by some V-jay and other artists.
I’m not familiar with this, but it looks very cool!
There are actually a lot of environments that are sort of along the same lines. A poster here has written a system like this in Lua that’s quite impressive, and as I mentioned, there’s the Smalltalk environments.
A lot of systems built along these lines are for graphics, which is fine, but I’m more interested in optimizing for the use-case of letting people who don’t identify as programmers do real work in a way that has graphical representation that makes sense to them.
A lot of people have specific, complex, idiosyncratic systems for organizing their every-day tasks that they’ve built on top of inflexible things like spreadsheets or collections of existing websites, and I’d like to cater to that kind of thing – making it possible for them to automate some of that stuff and have an easier time because they can produce a structure that’s a better fit.
Excel is actually the most successful programming environment in that regard. The builtin tracing (intermediate values stored in cells) and the spacial representation makes it approachable for people who think that they are not programmers. I think that this is amazing as those same people’s mind would often freeze if they assumed it was a programming language. It’s also deployed in countless of environment and powers a lot of business processes around the world.
But as you noticed, Excel only works for specific types of problems. So far, all the attempts to produce a generic visual programming language have been a failure in my opinion so maybe the solution is to keep the language specialized and then make many of them so all the use-cases can be covered. Create a bunch of UI DSLs.
Yeah. My prototype is not a visual programming language, but instead a framework built on top of a dialect of Io.
You may already be aware of this but here’s an example about composability, especially with respect to editing parents. Its not primarily “for making UIs”.
For each of these, which of your items are missing?
Can you post some screenshots and/or describe and discuss your current implementation, even if not final? It would help clarify some of your implicit assumptions about the problem and model. (For example, what are things you would consider a widget and not a widget.)
Why only one? Or do you mean at least one?
Also why only one? What about visually nested things? (Is a representation something already rendered or an abstract description?)
Edit: Also, can you post some links to some examples of UIs on TV you consider good examples of what you want to do?
I consider Smalltalk to have too high an initial learning overhead for this purpose (since my goal is to have the composability of graphical elements be an ‘invitation’ to programming in general for casual users).
Tk has a lower initial learning overhead but there are a lot of inconsistencies in how different widgets work (along with problems like a single-thread model), and no system has been written using Tk that does composability in the natural way smalltalk environments do. (I attempted a TCL/TK implementation many years ago. It’s briefly described here. It quickly became unmaintainable, and the widgets were not flexible enough for my liking.)
Sure!
Io is an extremely simple homoiconic language with a prototype-based object system. It looks a lot like rebol. I chose it because an important part of making a language look understandable to non-programmers is conceptual simplicity, and Io’s pointfree style and message-passing-like behavior is a good fit for a context where a lot of the coding is glue between objects.
Because of certain features I wanted that stock Io didn’t have and certain behaviors that might be bugs (and because Io is not maintained), I decided to write my own implementation in a language with primitives that are a better match for the parts of Io I consider important for this task.
Since Io has a prototype-object system and not a class-object system, exploration through cloning a working object and modifying its guts is possible at run time. I expect this to be a normal pattern for experimentation.
I have defined the ‘widget’ object as having a position, size, and bitmap. Any widget inherits these properties. So, any widget has a visual representation, in the sense that it has a bitmap & that bitmap gets blitted to the display at render time. (Of course, it can be transparent, or whatever.) If you look at my implementation, I’ve also got some logic for visually nested things: children are positions at offset with respect to their parents, always at a higher layer, and their position affects the computed size of their parent.
The importance of widgets having a visual representation is that the expected method of modifying a widget is by selecting it in the interface and inspecting its implementation. So, ideally, every live object would be a widget whose visual representation indicates its state. (I don’t plan to make non-widget objects impossible, but it’s bad form, since they’re going to be harder to find and change.)
Unfortunately, I haven’t gotten to the point of having anything render. Stock Io’s various graphics bindings won’t build for me, & my reimplementation has a ways to go. Screenshots might be misleading, though: my goal is to have a system that’s extremely mutable in its form. Squeak’s smalltalk-based desktop is a good representation of the kind of thing I’m aiming for.
I’ll dig up some. Alternately, TVTropes has some lists.
Think along the lines of the way viruses are depicted in the movie Hackers – as graphical objects with apparent properties. Obviously, viruses have other reasons for not exposing their nature graphically – the whole point of them is to be hidden from & hostile to the end user – but having a graphical representation that matches the end user’s mental model of a task (as opposed to skeuomorphism, wherein it matches a programmer’s model of a device that used to accomplish a similar task, or the current norm, where the UI maps onto the structure of the programmer’s code) is desirable, even though it can’t be attained unless the end user writes it themselves. (The goal here is to minimize the artificial friction users encounter in making their computing environment a natural reflection of their mind.)
Thanks, that’s a very interesting read and I understand better now.
Does this mean everything will mainly be raster (as opposed to vector). What about other visual transformations? How (un)important are they? Or maybe my question actually is whether composition will mainly be on functionality or visual appearance as well?
Hmm…what about overlays for viewing and editing widgets, like corners, text cursors and bounding boxes? I guess they could all be their own separate overlay widgets.
That might not be an issue for prototyping, except possibly for testing speed. To try it out, you might have a mock “render” loop that just prints out everything that should be drawn (and where) for that frame.
And some random questions about the implementation you linked too.
I’m a bit concerned about
calculateBboxandwidgetLint’s potential resource (time) consumption.For
setParent, shouldn’t the current parent, if there is one, also remove “self” as a child?What about user input? Do they just get passed up the tree from the child? What if you want to globally change keybindings afterwards? Does the prototype-object system help with this or makes it harder?
A lot of what I’ve written here is probably too early to think about (although something like vector vs raster is pretty hard to change later on).
I definitely agree with Squeak not being inviting enough. I haven’t used Tk all that much despite having made this but did find issues with its internal model a few times.
I guess if Io is simple enough, it can even be part of the thing that’s to be rewritten/customized.
I hope you go much further with these ideas!
In the particular system I’m writing, I’m only supporting raster, because I am lazy. I’m adopting the model from Menuet, because rendering becomes trivial: clear buffer, blit existing bitmaps onto buffer at given positions in Z order, flip. (I have the necessary information to later optimize out totally occuluded stuff, or keep a dirty bit in order to determine what parts of the display should be updated, if performance becomes a problem, but since unlike the Applet system in Java the user code doesn’t re-draw on visibility, it actually will be less of an issue, with the downside being the occasional half-drawn widget being rendered for one frame.)
Composition is performed through message-passing. My goal is to provide the GUI equivalent of unix pipes.
Yeah. It cleans the system up conceptually if there are no “special” classes of widgets that don’t count as widgets.
The stage I’m at in development is concerned with rewriting Io so that it can actually run my program.
This is a proof of concept system & so I’m not concerned with performance at all. If major performance problems appear, I’ll look at some of the low-hanging fruit (like the Io intepreter re-parsing and re-tokenizing the entire expression over and over) first.
Yes, from the focused child.
That’s not a facility I’m supporting, though generally speaking you’d only be creating key bindings for widgets that use them, so having layers of conflicting triggers would be bad form.
It depends on how you use it. The inheritance hierarchy is distinct from the widget-packing hierarchy, and both are used for message-handling (of which input handling is a part). It’s definitely possible to make a huge mess with this.
I actually made a lot of these decisions before I started writing any code, so you’re absolutely right to ask these questions. Because I expect to have between zero and one users, performance isn’t a priority, and I’m ignoring it in favor of making everything structurally & conceptually straightforward.
Io is pretty bare-bones and I’m looking to write as much as possible in it, as opposed to Tk, where complex things like the text entry widget are basically exposed from FFI. I probably won’t expose the actual parser to be rewritten, though.
I’m curious – has there been any development on this since? Would you be interested in any volunteer work? I’ve got some time to kill, as I don’t start work for a while. PM me if you’re interested.
I haven’t done any work on this for a while, no. I’d love somebody to pick up where I left off, or work on their own projects inspired by this.
I’m likely to work on it very intermittently. I have a lot of side projects, and with the current load at work, I get a chance to commit to one maybe once every few weeks. This one is complicated, since I’m writing a whole programming language interpreter for it, so it’ll probably get less love than my simpler projects.
I’ve started poking at it in https://github.com/tekknolagi/kamui (having filtered out the subfolder from your misc repository). I’m very new to Io and this idea, so we’ll see if I get anywhere at all.
One of the reasons why composable text based interfaces are so powerful is that they allow you to save your work as scripts. I can imagine a system where actions are saved as macros that could be composed together, but I don’t know if I can see it achieving the same fidelity as a text based interface. Is there value in opening up example based macros for composable interfaces? If so, could they be mixed and matched and pushed back into the widget to make more complex primitives? (E.g., like creating a new brush from two basic brushes.)
On the text side, I see the prototype-object facility for widgets providing an example-macro-like functionality. Unfortunately, outside of trivial stuff (like stealing the Alto’s method of representing a composition of two applications: the gesture of drawing an arrow between open windows), I’m not sure how to expose composability in a graphical way. I’m instead trying to make the jump to text as natural as possible.
I’m certainly no expert in GUI programming, but these guidelines seem to completely miss a major source of complexity: state management and duplication. The immediate mode GUI approach is the only one I’m aware of that minimizes state complexity, though Elm looks pretty close. My shallow understanding is that React provides some of the benefits of immediate mode, but not all of them.
Edit: for an example of how powerful immediate mode GUI programming can be, watch this handmade hero episode, where Casey implements a simple radial menu in under an hour: https://youtu.be/ftZIujU3Udw
In the system I’m proposing, the general rule of thumb is that all state should be visible.
The system will absolutely get hairy. I don’t think that matters much, so long as the hairiness is the product of the primary user & is interacted with frequently enough. In this context, I am opposed to scale.