The title is a little misleading as Oil is yet another shell, so it’s not so much “improving a Bash script using Oil” as it is “replacing a Bash script with an Oil script”.
That’s true, but you can also use Oil just to improve a bash script… just parsing or running it with Oil may make the script more comprehensible, and less reliant on the vagaries of bash (which do change from version to version). Example with a Lisp in bash:
The idea has come up quite a bit over the years, at varying levels of quality. https://github.com/huyng/bashmarks is the old version of it I’ve seen, but isn’t friendly to file paths with spaces.
I ended up making 3 versions of something similar for myself, one in bash, one in Go, and then I ported the database manager to Nim. I’ve started to see it as the shell equivalent of making your own blogging software, and a great way to make your shell suck a lot less.
Thanks for bringing warp to my attention. I didn’t know about it, goto indeed looks very similar to it. I am not familiar with zsh and I can’t fully compare them, I built goto for the Bash shell, its zsh functionality was added by contributors. If I find the time, I’ll give warp a try and come back to you.
This is dangerous, as you are now potentially telling people their name is not actually their name. You link to the Falsehoods Programmers Believe About Names in the article itself, but I would still like to point out number 12:
People’s names are case sensitive.
and number 13:
People’s names are case insensitive.
Trying to capitalise names based on what you believe to be correct does not guarantee it actually is correct. Telling someone called “Jack MCK” they are actually called “Jack McK” would be incorrect, but would be the result this script gives you. Serving them a page with their name incorrectly capitalised can be hurtful, especially because they did enter it correctly themselves.
That’s indeed what’s also said in the article itself. It notes that you might just use the snippet for suggesting a normalized format, letting a user decide which form is correct.
But why assume it’s incorrect in the first place? You would still be telling them “hey I believe your name is incorrect, do you actually know how to spell your own name?”.
Maybe I’m missing something, but I don’t see how this is the first draft of an HTML successor? Isn’t this just a meta language to abstract over HTML in its current form?
But in the future browser engines could just parse this directly. I also have one for CSS called Hakon, http://treenotation.org/sandbox/build/#hakon, and there are a few in progress for Javascript. So in the future you could have 1 syntax for all the languages. Your HTML IDs would leverage the same typing information used by your CSS and your Javascript–no more bugs due do having to duplicate that kind of information.
So I see this in the long term as the beginning of the successor to the HTML/CSS/JS stack.
That looks like a DTD, except with a Pythonesque syntax instead of SGML.
I don’t see where there’s value in it for HTML. It may be a neat macro language, but it seems quite presumptuous to call it a successor. HTML survived browser wars, pre-WHATWG useless specs, XHTML and XHTML2. It’s used for interoperability and compatibility with the massive amount of existing content. Such strong network effects are generally impossible to break with same-but-better things.
Thank you for the comments and all very fair points.
“HTML survived browser wars….”
HTML, created by Berners-Lee, came out around 1990. XHTML in 2000. So HTML is about 30 years old.
Binary Notation, as formulated by Leibniz, came out in 1689. So over 300 years old.
Tree Notation is more like “Binary Notation” than it is a programming language. Just as binary is the simplest way to encode information, Tree Notation is the simplest way to encode abstraction. You cannot improve upon it, and in trying to improve upon HTML/XML/Lisp/any other language people will keep reducing until they reach Tree Notation.
HTML survived for 30 years but I don’t think in 30 years we’ll be using it. We’ll have switched to Tree Languages. (Or perhaps, in 30 years will have switched to Tree Languages and then to something else even better that I can’t foresee).
“Such strong network effects”
Tree Notation will also have very strong network effects over time. A few years ago it was just an idea, but now in an hour you can create a whole new Tree language in Tree Notation and get a parser, compiler, VM, syntax highlighting, autocomplete, program synthesis, type checking, visualization, and a whole lot more.
“it seems quite presumptuous”
When you discover what is comparable to a new fundamental law of physics, and confirm your discovery over many years with immense amounts of data and experiments, you have 3 options: 1) don’t tell anyone, 2) tell people but lie and say it will not cause a massive paradigm shift 3) tell people the truth and be accused of madness/arrogance/presumptuousness. I’ve decided option 3 is the lesser of 3 evils. It’s a long and lonely road but a few people have been immensely helpful and we are making great progress!
Reminds me of the kind of language used in all those advertisements for quack medicinal products in centuries past.
HTML survived for 30 years but I don’t think in 30 years we’ll be using it. We’ll have switched to Tree Languages.
This is so incredibly off-base it verges on delusion.
in an hour you can create a whole new Tree language in Tree Notation and get a parser, compiler, VM, syntax highlighting, autocomplete, program synthesis, type checking, visualization, and a whole lot more.
I don’t think anyone would want to use them, and it’s not possible to generalise the creation of usefully distinguishable VMs.
There are no actual evils in your three options, but the third will certainly lead to people ignoring you. If you consider the opposite of evil to be people enjoying and appreciating your creation, then it’s definitely not the lesser of the three; try option 2.
Reminds me of the kind of language used in all those advertisements for quack medicinal products in centuries past.
I think this is a very healthy attitude to have. Probably 99/100 times someone says something like that they are full of it, so it is safe to ignore them. I’m not asking you to believe. I’m asking people to try your best to prove me wrong.
Can you improve upon binary notation? Can you come up with a simpler way to represent all information than just using 2 markings? If you can, you may win a Nobel or Turing Prize. If not, then it shouldn’t be too controversial to say that binary notation is the best notation for storing information.
Likewise, can you improve upon tree notation? Can you come up with a simpler way to represent all abstractions than just using horizontal and vertical spacing? I can’t, have been trying for 7 years, and at this point have code that turns more than 10,000 different languages into Tree Notation, covering hundreds of languages concepts. If you can figure out something better, please, please, share it with me so I can move on to other things and you may win a Nobel or Turing Prize.
This is so incredibly off-base it verges on delusion.
You are right! E pur si muove!
I don’t think anyone would want to use them
This is a great criticism, and my only answer is we are making progress! The Tree Language powering Ohayo is getting quite good, and could be the first killer language, in a few weeks to months. I did underestimate (by 10x+) the amount of infrastructure and data we needed to build before we had killer-app Tree Languages. It is not as simple as coming up with a set of words that compiles to the language of interest. That does not get you a great language. Even though the base of Tree Notation is so simple, you still need to think through dozens to hundreds of design decisions to make a great language. Luckily, we’re automating a ton of that through a huge data gathering and machine learning effort. More to show on that front in 6 - 9 months.
it’s not possible to generalise the creation of usefully distinguishable VMs.
This is a good insight. However, we can indeed make the building of “usefully distinguishable VMs” 10x easier through both the rules of the language itself coupled with a huge dataset of languages, machines, and patterns.
There are no actual evils in your three options,
Yes, there are.
If I tell no one, people die, diseases remain harder to cure, and a whole host of other problems remain more difficult because we are wasting incredible amounts of time and energy using suboptimal tools.
If I tell people but say “this only applies to some small domain”, not only am I going against my code of being honest, but we miss out on the huge network affects that come from people utilizing the universality of Tree Notation, leading again to problem #1 above.
Personally it’s not fun to invest so much of my time and resources into this, but there’s nothing I can do more that would have a more positive impact on the world than this. Until other people “get it”, and the community grows, I feel a responsibility to keep pushing the ball forward.
the third will certainly lead to people ignoring you.
People ignore Tree Notation (and me) no matter what. I’m not worried about the majority of people. They’ve got their own lives to worry about. My strategy is to reach people who understand “exponential thinking”. If there’s a 1% chance I’m right, and let’s say software productivity goes up 30% in 10 years. We are talking about an annual value in the trillions. Even at 99% odds of me being wrong this is a many-billion-dollars idea in expected value for the world.
One negative side-effect you may be missing is that people might ignore you and tree notation who would otherwise be interested. E.g. I’m curious about the word “bern” in your example – does that have meaning? But I’m guarded against engaging and asking because I’m worried I’ll have a crazy person following me around and not going away.
I’m curious about the word “bern” in your example – does that have meaning?
It’s a placeholder for that keyword that I added a long time ago and haven’t revisited. At the time I couldn’t think of a word for “content” that wasn’t so generic and so went with a unique word that will be easy to find/replace later. Looking at it now my first though was a reference to Bernie Sanders, but then looking back I realized it was a nod to Tim Berners-Lee, which makes more sense. Anyway, I added a todo to the Grammar file for Stump with a clarification there.
But I’m guarded against engaging and asking because I’m worried I’ll have a crazy person following me around and not going away.
Haha this made me laugh out loud. (And then cry a little bit inside.)
It is true though. I strive to be a good scientist. Marketing and community organizing are not my thing. I often jump on the chance to have a conversation about an exponential phenomenon which online can come across as crazy (in person I can quickly assess if someone wants to talk exponential or the more calm here and now). The best decision I made at my last startup was removing myself from the CEO role and recruiting a guy to take over the business and marketing so I could just focus on the code and science. With the Tree Notation Lab I started, I hope to do the same–find a good manager and a good community organizer to lead the “people” efforts so I can focus on the science.
You cannot improve upon it, and in trying to improve upon HTML/XML/Lisp/any other language people will keep reducing until they reach Tree Notation.
There have been many proposals over the years for “improving” upon s-expression notation for Scheme and Lisp, including indentation-based “tree notation”. This has never caught on. In the Ruby/Rails scene there has been HAML for a long time, which is basically this notation. That never really became massively embraced by the Rails community either. Sass did catch on massively, even outside the Ruby community, but only after it grew a more traditional CSS-like syntax.
Given this history, I have no reason to believe yet another indentation-based notation for HTML is going to change the world.
Besides, there are downsides to indentation-based notation for communication. I experience this several times a week when I try to copy/paste some Python code and the indentation gets lost in translation. At least with s-expressions the layout may be messy, but it’ll still work when you copy and paste stuff.
There have been many proposals over the years for “improving” upon s-expression notation for Scheme and Lisp, including indentation-based “tree notation”
I currently track 38 of these, which is a shockingly small number of languages (~99% of languages are not indentation-based). The one closest to Tree Notation is I-Expressions, created in 2003 by Egil Möller. It has not gained traction, yet.
The difference between Tree Notation and these, like HAML/SASS/etc, are network effects. Currently if I write a tool for editing HTML, it only helps HTML authors. But if I write a tool for editing a Tree Language that targets HTML, it also helps people using Tree Notation to target assembly, C++ , TypeScript, Java, et cetera.
Most of my time goes into work on some new very high level Tree languages. All of the syntax highlighting, autocomplete, error suggestions, etc, that are now present in this Stump HTML language, weren’t built for that at all. That’s why these indentation languages will be different than the 38 that have come before.
Another nice catch! Thanks. Flipped that bit in my DB. Looking at this now, I’m realizing that my crawler/ML code was not pulling out indentation languages and that bit was only getting set manually over the years.
This list is missing sweet-expressions aka SRFI 110 aka t-expressions, which is an evolution on i-expressions (be sure to read the design rationale, it has some criticism of i-expressions you might want to consider).
The page is locked behind a paywall
True. Should I not have posted it? Or, should I have mentioned the paywall in the comment? (Thanks…)
As a rule people don’t post paywalled content here, because it’s impossible to discuss.
Right; thanks.
If it’s any consolation, the email sent out by lobsters has the full article. I can post it somewhere if you want.
I was hoping to stimulate a technical discussion of the topic, so yes!
This is a direct save from my email client, which is why there’s all those weird characters. I’m a bit too busy to clean up the text. But, hey, at least it’s better than nothing: https://gist.github.com/lattera/903f0ede4a27506de40787528459ddcd
The title is a little misleading as Oil is yet another shell, so it’s not so much “improving a Bash script using Oil” as it is “replacing a Bash script with an Oil script”.
That’s true, but you can also use Oil just to improve a bash script… just parsing or running it with Oil may make the script more comprehensible, and less reliant on the vagaries of bash (which do change from version to version). Example with a Lisp in bash:
http://www.oilshell.org/blog/2020/06/release-0.8.pre6.html#patch-to-run-the-mal-lisp
Interesting project! It seems like a very similar system to warp directory, what are the advantages of using
gotoinstead ofwd?The idea has come up quite a bit over the years, at varying levels of quality. https://github.com/huyng/bashmarks is the old version of it I’ve seen, but isn’t friendly to file paths with spaces.
I ended up making 3 versions of something similar for myself, one in bash, one in Go, and then I ported the database manager to Nim. I’ve started to see it as the shell equivalent of making your own blogging software, and a great way to make your shell suck a lot less.
Thanks for bringing warp to my attention. I didn’t know about it, goto indeed looks very similar to it. I am not familiar with zsh and I can’t fully compare them, I built goto for the Bash shell, its zsh functionality was added by contributors. If I find the time, I’ll give warp a try and come back to you.
This is dangerous, as you are now potentially telling people their name is not actually their name. You link to the Falsehoods Programmers Believe About Names in the article itself, but I would still like to point out number 12:
and number 13:
Trying to capitalise names based on what you believe to be correct does not guarantee it actually is correct. Telling someone called “Jack MCK” they are actually called “Jack McK” would be incorrect, but would be the result this script gives you. Serving them a page with their name incorrectly capitalised can be hurtful, especially because they did enter it correctly themselves.
That’s indeed what’s also said in the article itself. It notes that you might just use the snippet for suggesting a normalized format, letting a user decide which form is correct.
But why assume it’s incorrect in the first place? You would still be telling them “hey I believe your name is incorrect, do you actually know how to spell your own name?”.
Maybe I’m missing something, but I don’t see how this is the first draft of an HTML successor? Isn’t this just a meta language to abstract over HTML in its current form?
Today this compiles to HTML.
But in the future browser engines could just parse this directly. I also have one for CSS called Hakon, http://treenotation.org/sandbox/build/#hakon, and there are a few in progress for Javascript. So in the future you could have 1 syntax for all the languages. Your HTML IDs would leverage the same typing information used by your CSS and your Javascript–no more bugs due do having to duplicate that kind of information.
So I see this in the long term as the beginning of the successor to the HTML/CSS/JS stack.
That looks like a DTD, except with a Pythonesque syntax instead of SGML.
I don’t see where there’s value in it for HTML. It may be a neat macro language, but it seems quite presumptuous to call it a successor. HTML survived browser wars, pre-WHATWG useless specs, XHTML and XHTML2. It’s used for interoperability and compatibility with the massive amount of existing content. Such strong network effects are generally impossible to break with same-but-better things.
Thank you for the comments and all very fair points.
HTML, created by Berners-Lee, came out around 1990. XHTML in 2000. So HTML is about 30 years old.
Binary Notation, as formulated by Leibniz, came out in 1689. So over 300 years old.
Tree Notation is more like “Binary Notation” than it is a programming language. Just as binary is the simplest way to encode information, Tree Notation is the simplest way to encode abstraction. You cannot improve upon it, and in trying to improve upon HTML/XML/Lisp/any other language people will keep reducing until they reach Tree Notation.
HTML survived for 30 years but I don’t think in 30 years we’ll be using it. We’ll have switched to Tree Languages. (Or perhaps, in 30 years will have switched to Tree Languages and then to something else even better that I can’t foresee).
Tree Notation will also have very strong network effects over time. A few years ago it was just an idea, but now in an hour you can create a whole new Tree language in Tree Notation and get a parser, compiler, VM, syntax highlighting, autocomplete, program synthesis, type checking, visualization, and a whole lot more.
When you discover what is comparable to a new fundamental law of physics, and confirm your discovery over many years with immense amounts of data and experiments, you have 3 options: 1) don’t tell anyone, 2) tell people but lie and say it will not cause a massive paradigm shift 3) tell people the truth and be accused of madness/arrogance/presumptuousness. I’ve decided option 3 is the lesser of 3 evils. It’s a long and lonely road but a few people have been immensely helpful and we are making great progress!
Reminds me of the kind of language used in all those advertisements for quack medicinal products in centuries past.
This is so incredibly off-base it verges on delusion.
I don’t think anyone would want to use them, and it’s not possible to generalise the creation of usefully distinguishable VMs.
There are no actual evils in your three options, but the third will certainly lead to people ignoring you. If you consider the opposite of evil to be people enjoying and appreciating your creation, then it’s definitely not the lesser of the three; try option 2.
I think this is a very healthy attitude to have. Probably 99/100 times someone says something like that they are full of it, so it is safe to ignore them. I’m not asking you to believe. I’m asking people to try your best to prove me wrong.
Can you improve upon binary notation? Can you come up with a simpler way to represent all information than just using 2 markings? If you can, you may win a Nobel or Turing Prize. If not, then it shouldn’t be too controversial to say that binary notation is the best notation for storing information.
Likewise, can you improve upon tree notation? Can you come up with a simpler way to represent all abstractions than just using horizontal and vertical spacing? I can’t, have been trying for 7 years, and at this point have code that turns more than 10,000 different languages into Tree Notation, covering hundreds of languages concepts. If you can figure out something better, please, please, share it with me so I can move on to other things and you may win a Nobel or Turing Prize.
You are right! E pur si muove!
This is a great criticism, and my only answer is we are making progress! The Tree Language powering Ohayo is getting quite good, and could be the first killer language, in a few weeks to months. I did underestimate (by 10x+) the amount of infrastructure and data we needed to build before we had killer-app Tree Languages. It is not as simple as coming up with a set of words that compiles to the language of interest. That does not get you a great language. Even though the base of Tree Notation is so simple, you still need to think through dozens to hundreds of design decisions to make a great language. Luckily, we’re automating a ton of that through a huge data gathering and machine learning effort. More to show on that front in 6 - 9 months.
This is a good insight. However, we can indeed make the building of “usefully distinguishable VMs” 10x easier through both the rules of the language itself coupled with a huge dataset of languages, machines, and patterns.
Yes, there are.
People ignore Tree Notation (and me) no matter what. I’m not worried about the majority of people. They’ve got their own lives to worry about. My strategy is to reach people who understand “exponential thinking”. If there’s a 1% chance I’m right, and let’s say software productivity goes up 30% in 10 years. We are talking about an annual value in the trillions. Even at 99% odds of me being wrong this is a many-billion-dollars idea in expected value for the world.
One negative side-effect you may be missing is that people might ignore you and tree notation who would otherwise be interested. E.g. I’m curious about the word “bern” in your example – does that have meaning? But I’m guarded against engaging and asking because I’m worried I’ll have a crazy person following me around and not going away.
It’s a placeholder for that keyword that I added a long time ago and haven’t revisited. At the time I couldn’t think of a word for “content” that wasn’t so generic and so went with a unique word that will be easy to find/replace later. Looking at it now my first though was a reference to Bernie Sanders, but then looking back I realized it was a nod to Tim Berners-Lee, which makes more sense. Anyway, I added a todo to the Grammar file for Stump with a clarification there.
Haha this made me laugh out loud. (And then cry a little bit inside.)
It is true though. I strive to be a good scientist. Marketing and community organizing are not my thing. I often jump on the chance to have a conversation about an exponential phenomenon which online can come across as crazy (in person I can quickly assess if someone wants to talk exponential or the more calm here and now). The best decision I made at my last startup was removing myself from the CEO role and recruiting a guy to take over the business and marketing so I could just focus on the code and science. With the Tree Notation Lab I started, I hope to do the same–find a good manager and a good community organizer to lead the “people” efforts so I can focus on the science.
There have been many proposals over the years for “improving” upon s-expression notation for Scheme and Lisp, including indentation-based “tree notation”. This has never caught on. In the Ruby/Rails scene there has been HAML for a long time, which is basically this notation. That never really became massively embraced by the Rails community either. Sass did catch on massively, even outside the Ruby community, but only after it grew a more traditional CSS-like syntax.
Given this history, I have no reason to believe yet another indentation-based notation for HTML is going to change the world.
Besides, there are downsides to indentation-based notation for communication. I experience this several times a week when I try to copy/paste some Python code and the indentation gets lost in translation. At least with s-expressions the layout may be messy, but it’ll still work when you copy and paste stuff.
I currently track 38 of these, which is a shockingly small number of languages (~99% of languages are not indentation-based). The one closest to Tree Notation is I-Expressions, created in 2003 by Egil Möller. It has not gained traction, yet.
The difference between Tree Notation and these, like HAML/SASS/etc, are network effects. Currently if I write a tool for editing HTML, it only helps HTML authors. But if I write a tool for editing a Tree Language that targets HTML, it also helps people using Tree Notation to target assembly, C++ , TypeScript, Java, et cetera.
Most of my time goes into work on some new very high level Tree languages. All of the syntax highlighting, autocomplete, error suggestions, etc, that are now present in this Stump HTML language, weren’t built for that at all. That’s why these indentation languages will be different than the 38 that have come before.
I’d be very interested in the references you mention, do you have a document somewhere that lists them?
I don’t have it online yet (I’m slowly getting around to open sourcing my programming language database), but here they are as CSV:
There’s also wisp, which seems to be a separate project from sweet-expressions.
Another nice catch! Thanks. Flipped that bit in my DB. Looking at this now, I’m realizing that my crawler/ML code was not pulling out indentation languages and that bit was only getting set manually over the years.
This list is missing sweet-expressions aka SRFI 110 aka t-expressions, which is an evolution on i-expressions (be sure to read the design rationale, it has some criticism of i-expressions you might want to consider).
Good catch, thank you. I track Sweet Expressions but mistakenly omitted the hasSemanticIndentation? flag.
I actually think Sweet Expressions were a regression on I-Expressions.
Hakon doesn’t work correctly. Try setting border to value “1px solid red”.
I’m probably missing something, but this looks more like a proof of concept than a proof of an idea to me.
I wrote this in Hakon:
It compiled to this:
Is that incorrect?