As a lisper, I would disagree with the basic premise that “shells” or programming environments cannot have a good REPL and be a good programming language.
The solution is that we have one tool, but there are two things, and so there should be two tools.
At risk of being obvious; “Things are the way they are for reasons.” These two tools will exist on day 1 and on day 2 somebody will make sure both of them are scriptable, because being scriptable is the killer feature of shells. It’s what makes shells useful. Even DOS had a scriptable shell, it’s that great a feature.
As a lisper, I would disagree with the basic premise that “shells” or programming environments cannot have a good REPL and be a good programming language.
The original article says that (good?) programming languages require “readable and maintainable syntax, static types, modules, visibility, declarations, explicit configuration rather than implicit conventions.”
As a non-lisper, I’ve never found Lisp syntax readable or maintainable. The books I’ve read, and the Lispers I’ve known, all swore that it would all click for me at some point, but nope. I get lost in all the parentheses and identical looking code.
As a random example, here’s a function I found online that merges two lists:
(defun myappend (L1 L2)
(cond
; error checking
((not (listp L1)) (format t "cannot append, first argument is not a list~%" L1))
((not (listp L2)) (format t "cannot append, second argument is not a list~%" L2))
; base cases
((null L1) L2)
((null L2) L1)
; general case, neither list is empty
(t (cons (car L1) (myappend (cdr L1) L2)))))
I would put out my eyes if I had to read code like that for more than a day. The last line with its crystal clear ))))) is the kicker. I know that this may be (largely? entirely?) subjective, but I think it’s a relatively common point of view.
My problem with that code isn’t the parens per se. It’s that you as the reader have to already know or be able to infer what is evaluated and what is not. defun is a primitive function and it’s really being called, but myappend and L1 and L2 are just symbols being defined. cond is a function and is really called, but the arguments to cond are not evaluated. However, cond does execute the first item of each list it is past and evaluate that to figure out which branch is true. Presumably cond is lazy only evaluates until it reaches its first true condition. format I guess is a function, but I have no idea what t is or where it comes from. (Maybe it’s just true?) null is a primitive symbol, but in this case, maybe it’s not a value, it’s a function that tests for equality? Or maybe cond handles elements with two items differently than elements with one item? cons, car, and cdr are functions and they are really evaluated when their case is true…
Anyhow, you can work it out with some guesswork and domain knowledge, but having syntax that distinguishes these things would be much more clear:
defun :myappend [:L1 :L2] {
cond [
[{not (listp L1)} {format t "cannot append, first argument is not a list~%" L1}]
[{isNull L1} {L2}]
[{isNull L2} {L1}]
[{true} {cons (car L1) (myappend (cdr L1) L2)}]
]
}
I don’t love the parentheses, but they’re not my biggest stumbling block with Lisp either. I find it hard to read Lisp because everything seems the same—flat somehow. Maybe that’s because there’s so little visual, explicit syntax. In that case, (maybe?) we’re back to parentheses. I’m not sure.
It’s because you’re reading the abstract syntax tree almost literally. In most languages, syntactic patterns create structure, emphasis, guidance. What that structure costs is that it can be a wall in your way when you’d prefer to go against the grain. In Lisp, there is almost no grain or limitation, so you have surprising power. In exchange, structure is your job.
Thanks: you’ve described the flatness (and its cause) much better than I could have. (I also appreciate your adding a benefit that you get from the lack of given or required structure.)
def myappend(l1, l2):
if not isinstance(list, l1):
pass
elif …
nothing surprising with “myappend”, “l1” and “l2” and no need for a different syntax^^
cond is a macro. Rightly, it’s better to know that, but we can infer it given the syntax. So, indeed, its arguments are not evaluated. They are processed in order as you described.
(format t "~a arg) (for true indeed) prints arg to standard output. (format nil …) creates a string.
So, I don’t know that Lisp would necessarily click for any given person, but I did find that Janet clicked things in my brain in a way few languages have since.
I built a lot of fluency with it pretty quick, in part because it was rather small, but had most of what I wanted I’m a scripting language.
That doesn’t apply to Common Lisp, though, which is both large and with more than a few archaic practices.
That being said, a lot of folks bounce off, and a lot of the things that used to be near exclusive to Lisp can be found elsewhere. For me, the simplicity of a smaller, postmodern parens language is the appeal at this point (I happen to be making my own, heh)
subjective. BTW Lisp ticks the other boxes. Personal focus on “maintainable”.
To merge two lists: use append.
Here’s the code formatted with more indentation (the right way):
(defun myappend (L1 L2)
(cond
;; error checking
((not (listp L1))
(format t "cannot append, first argument is not a list~%" L1))
((not (listp L2))
(format t "cannot append, second argument is not a list~%" L2))
;; base cases
((null L1)
L2)
((null L2)
L1)
;; general case, neither list is empty
(t
(cons (car L1) (myappend (cdr L1) L2)))))
Did you learn the cond macro? Minimal knowledge is required to read Lisp as with any other language.
Sure, but what I posted is (pretty obviously) teaching code. The function is called myappend because (presumably) the teacher has told students, “Here is how we might write append if it didn’t exist.”
Here’s the code formatted with more indentation (the right way)
Thank you, but that did nothing to make the code more readable to me. (See below on “to me.”)
subjective…Did you learn the cond macro? Minimal knowledge is required to read Lisp as with any other language.
I’m not sure, but you seem to be using “subjective” as a way of saying “wrong.” Or, to put this in another way, you seem to want to correct my (and Carl’s) subjective views. As a general rule, I don’t recommend that, but it’s your choice.
I’m glad you enjoy Lisp and find it productive. But you may have missed my point. My point was that I—and many people—do not find List readable or maintainable. (It’s hard to maintain what you find unpleasant and difficult to read, after all.) I wasn’t saying you or anyone else should change your subjective view; I was just stating mine. To quote jyx, “Yes, feeling is subjective and is fine.” I didn’t mean to pick on something you love. I was questioning how widely Lisp can play the role of great REPL plus readable maintainable programming language.
hey, right, I now find my wording a bit rigid. My comment was more for other readers. We read a lot of lisp FUD, so sometimes I try to show that the Lisp world is… normal, once you know a few rules (which some busy people expect to know when they know a C-like language).
To merge two lists: use append.
rewording: “dear newcomer, be aware that Lisp also has a built-in for this”. I really mean to say it, because too often we see weird, teaching code, that does basic things. Before, these examples always bugged me. Now I give hints to my past self.
My point was that I—and many people—do not find List readable
OK, no pb! However I want to encourage newcomers to learn and practice a little before judging or dismissing the language. For me too it was weird to see lisp code at the beginning. But with a little practice the syntax goes away. It’s only syntax, there is so much more to judge a language. I wish we talked less about parens, but this holds for any other language when we stop at the superficial syntax.
or maintainable
but truly, despite one’s tastes, Lisp is maintainable! The language and the ecosystem are stable, some language features and tooling explicitly help, etc.
They must do this because they deploy different components at different times. When you do apt-get upgrade, components are selectively replaced without recompilation. (Compilation is the thing that enforces static types!)
Windows does the same thing. When you do a Windows update, it replaces DLLs. COM and whatever the modern equivalent is are dynamic, not static. You query for interfaces at runtime.
So static typing doesn’t scale even to a single machine – on either Linux or Windows – let alone multiple machines, which is where I use shell a lot. (OS X or Windows talking to Linux is another very common use of shell!)
So shell has the same issue as C/C++ – components can be replaced independently without global versioning, and you need to evolve in a compatible way. (BTW there used to be this thing called Linux FHS, which would solve some analogous issues for shell as ABIs do for C++ – https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard – however it doesn’t seem to have reached its goals)
So basically, if you have a statically typed shell, you could only use it with a Google-like monorepo with a global version number and static linking.
Though even Google can’t redeploy every binary in a service at once, which is why protobufs have optional fields queried at runtime, not compile time. e.g. see Maybe Not by Hickey: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey)
So Google would still need a dynamically typed shell. Also, feature flags are the idiomatic way to upgrade services there as well. Yes, literal command line flags to server binaries, even though they have a monorepo and protobuf definitions.
So shells aren’t statically typed for the same reason that the Internet isn’t statically typed: you can’t upgrade everything at once. You don’t download the latest version of TCP/IP and link it into your programs like you do a gRPC definition. Instead you use the protocols dynamically (binary in the case of TCP/IP; text in the case of HTTP, HTML, JSON, etc.)
OP is talking about static types in the language level, and you seem to be talking about static vs dynamic linking, which … I think are kind of almost unrelated?
A shell is intrinsically a dynamic linker, just not in the sense that the term is routinely used. The purpose of a shell is to run other programs. Those other programs have interfaces (command-line flags, supported input and output) that change over time. You cannot map them into a static type system without also baking in the supported version. New tools are installed more frequently than a shell is upgraded and add to the space of shell types.
The .NET team had some interesting ideas about adding a section to binaries that described the tool’s interfaces and allowed them to be surfaced in PowerShell, but even then a static type system can express only the tools installed when the program is written.
Well outlined! Classical example of an hypothetical mystical epiphany vs reality. For 16 years now, every time discussions about shells come up, there goes the compulsory reference on how PowerShell is one step ahead and how going that path would materialize the utopia.
static typing doesn’t scale even to a single machine – on either Linux or Windows – let alone multiple machines, which is where I use shell a lot
https://www.unison-lang.org/ tries to scale static typing to internet level. They avoid compatibility problems using novel method: code (and types) stored in binary form in content addressable storage (I think this is Merkle tree). If old code (and types) are used, they are not deleted and are kept around forever. (Unless you stop to use them, then they are AFAIK garbagge-collected.)
Unison definitely has some cool ideas, and I’m glad someone is trying it
But it’s very far from how computers actually work today, and ironically their theory is to NOT have a shell at all, whether statically typed or not.
They’ll certainly have to solve open computer science problems in order to scale to even what a single Windows or Linux machine has today – e.g. ~30M to 100M lines of code. Let alone writing something like Google Maps, Twitter, etc. I’d guess they’re 100x to 1000x slower right now, or maybe exponentially slower, though I’d be interested in some real data
Very good points but I think general-purpose programming languages are simply not only not optimized for terseness, they’re usually incapable of being terse-but-still-descriptive.
My shell scripts are most often just wrappers that fix paths or provide default args or chain some calls. If a language (which is not Scala or Haskell) could be made to behave like that, e.g. by doing something like this:
import shell_stuff as *
x = open("name").grep("foo").sort("-r")
y = ...
but the margins are razor thin. It all breaks down if I have to do
I’ve been banging this drum for years, I’m happy to add another great post to the pile of examples for when people ask me “why do you hate shell scripts”!
Another point to add is that shell scripts, or any scripts really, are fairly personal. Not that all programming is somewhat personal, but scripts especially. So seeing an entire OS propped up by these incomprehensible-to-all-but-one stringly-typed soups of seemingly arbitrary characters would make me feel uneasy too if that was my daily-driver.
And of course on the flip-side, one could argue it’s better to see the soup than not be sure if it even is soup behind those closed source .dll files on my system… but ignorance is bliss!
I’m perfectly happy with fish for (1), interactive use. But I want a better shell for scripting. I’m hoping Oil will succeed there. At work I write lots of one-off scrips to soft transition codebases. The script might evolve into a complicated 10 step process that automates as much as possible while requiring input from me at certain steps. Usually I end up combining many different tools: awk, rg, fastmod, python, etc. Some parts might need to be parallelized and use the filesystem for mutual exclusion / synchronization. This kind of thing would be so much easier with a shell that has actual data structures.
The proposed solution of an API with a thinly wrapped auto-generated CLI is not terrible. I have heard it is common within Google, for example.
In the Python world, there are various solutions starting from plac/argh and moving on to Click/autocommand and probably N others.
Personally, I prefer Nim to Python which has cligen. As mentioned here, but not in the article, the overhead of dispatch to a program in shell REPLs can also be thousands to millions of times higher.
A statically linked lua without readline is smaller than most shells. I wish more systems would use it as their default scripting environment, rather than writing shell (or, worse, bash) scripts for complex tasks. Lua isn’t the best language in the world, but it’s far from the worst.
Lua plus dedicated additions to the standard library (e.g., expanded filesystem operations and string support) would be heaven as a replacement for scripting in shell.
I recently started using Lua for scripting a thing in a videogame. I find it so weird to be using a language which has advanced features like coroutines but also doesn’t have an equivalent of string.slice() in the standard library.
I find it so weird to be using a language which has advanced features like coroutines but also doesn’t have an equivalent of string.slice() in the standard library.
The standard library is very small. I wrote a split module for Lua, since that’s something that I find essential which isn’t part of Lua’s standard library. I think that some hosts expand Lua’s standard library to help out (and to suit the needs of their users).
In fairness, Lua is designed to be used as an embedded language. But I wish there were an optional “batteries” package. That said, LuaRocks is often helpful.
Maybe the solution is to go the other way? Some sort of OpenAPI like spec that allows you to define an API for a command - but the result is still a command line call, and the spec doesn’t have to be attached to the command but can be shipped with the package in a designated folder/format. That way, the people writing the shell can do the work of integrating commands that don’t support this format.
Have you seen powershell? It sounds a lot like what you describe. I mean, they have .NET reflection instead of “OpenAPI”, but I believe it implements what you describe.
There are several problems with the reverse direction.
Command help rarely indicates any more specific prog.lang types than string.
You may get number/int/float granularity, but, it’s rough. Meanwhile, prog.langs
can often both print&parse all kinds of rich types like enums, lists, sets, tables, ..
Commands occasionally also have their own bespoke home grown syntaxes mucking up the works. E.g., gcc differs from GNU ls.
Launching a command (new address space, new kernel scheduling entity, etc.)
is intrinsically much more costly.
Your OpenAPI/spec idea could help with 1), but it would be hard to auto-infer from --help-like docs. That means a lot of work on API wrapper writers (much like work for “TAB completion systems”, only maybe more so). Another way to think of 2) is that the “forward direction” (generating a CLI for an API) lowers the syntactic entropy/chaos of the world (by using just 1 toolkit/CLI syntax) while the reverse direction must confront it.
My impression from Gentoo, Nix etc. is that if you can get a few nerds interested in your system, the amount of effort they’re willing to put in to create compatibility can be much greater than the sum total of effort you can get upstream to commit to supporting you.
That’s why I think it’s more promising to keep the glue layer external.
(Though I also love the idea of in-language shell support.)
LOL. The Zsh completion people indeed put a lot of effort in! There may be other technical difficulties that maybe the .NET team published details about. I think a lot of it is just the natural language processing trying to go “from docs to parse types”. http://docopt.org/ is another effort along that arc which might have insights. There may also be an implied requirement that the target prog.lang have sum types, e.g. if the command can parse a parameter where “digits mean one thing and text another”.
I would like to see better libraries in Python and Go that make it super easy to run commands, connect their pipes to each other, check error values, read results, etc. Perhaps using shell-like syntax.
While this doesn’t solve the problem I love using the Elvish shell. It’s not statically typed but it has enough programming language features such as lists, maps, and builtin functions that I can use it for the vast majority of my scripts.
As a lisper, I would disagree with the basic premise that “shells” or programming environments cannot have a good REPL and be a good programming language.
At risk of being obvious; “Things are the way they are for reasons.” These two tools will exist on day 1 and on day 2 somebody will make sure both of them are scriptable, because being scriptable is the killer feature of shells. It’s what makes shells useful. Even DOS had a scriptable shell, it’s that great a feature.
The original article says that (good?) programming languages require “readable and maintainable syntax, static types, modules, visibility, declarations, explicit configuration rather than implicit conventions.”
As a non-lisper, I’ve never found Lisp syntax readable or maintainable. The books I’ve read, and the Lispers I’ve known, all swore that it would all click for me at some point, but nope. I get lost in all the parentheses and identical looking code.
As a random example, here’s a function I found online that merges two lists:
I would put out my eyes if I had to read code like that for more than a day. The last line with its crystal clear
)))))is the kicker. I know that this may be (largely? entirely?) subjective, but I think it’s a relatively common point of view.Admittedly there are a lot more people who routinely see
and feel everything is fine.
Yes, feeling is subjective and is fine.
you are kind. It is often:
You have an extra
;in there!I’d not let that pass in a code review 😜
Completely subjective and entirely valid.
My problem with that code isn’t the parens per se. It’s that you as the reader have to already know or be able to infer what is evaluated and what is not.
defunis a primitive function and it’s really being called, butmyappendandL1andL2are just symbols being defined.condis a function and is really called, but the arguments tocondare not evaluated. However,conddoes execute the first item of each list it is past and evaluate that to figure out which branch is true. Presumablycondis lazy only evaluates until it reaches its first true condition.formatI guess is a function, but I have no idea whattis or where it comes from. (Maybe it’s justtrue?)nullis a primitive symbol, but in this case, maybe it’s not a value, it’s a function that tests for equality? Or maybecondhandles elements with two items differently than elements with one item?cons,car, andcdrare functions and they are really evaluated when their case is true…Anyhow, you can work it out with some guesswork and domain knowledge, but having syntax that distinguishes these things would be much more clear:
This is something Clojure improves somewhat upon traditional Lisps, it uses
[]rather than()for grouping (see defn]).I don’t love the parentheses, but they’re not my biggest stumbling block with Lisp either. I find it hard to read Lisp because everything seems the same—flat somehow. Maybe that’s because there’s so little visual, explicit syntax. In that case, (maybe?) we’re back to parentheses. I’m not sure.
It’s because you’re reading the abstract syntax tree almost literally. In most languages, syntactic patterns create structure, emphasis, guidance. What that structure costs is that it can be a wall in your way when you’d prefer to go against the grain. In Lisp, there is almost no grain or limitation, so you have surprising power. In exchange, structure is your job.
Thanks: you’ve described the flatness (and its cause) much better than I could have. (I also appreciate your adding a benefit that you get from the lack of given or required structure.)
nothing surprising with “myappend”, “l1” and “l2” and no need for a different syntax^^
condis a macro. Rightly, it’s better to know that, but we can infer it given the syntax. So, indeed, its arguments are not evaluated. They are processed in order as you described.(format t "~a arg)(for true indeed) printsargto standard output.(format nil …)creates a string.we can replace
carwithfirstandcdrwithrest.So, I don’t know that Lisp would necessarily click for any given person, but I did find that Janet clicked things in my brain in a way few languages have since.
I built a lot of fluency with it pretty quick, in part because it was rather small, but had most of what I wanted I’m a scripting language.
That doesn’t apply to Common Lisp, though, which is both large and with more than a few archaic practices.
That being said, a lot of folks bounce off, and a lot of the things that used to be near exclusive to Lisp can be found elsewhere. For me, the simplicity of a smaller, postmodern parens language is the appeal at this point (I happen to be making my own, heh)
some things yes, but never all together, let alone the interactivity of the image-based development!
I mean, I think Factor is an example of of all of that coming together in a something distinctly not a Lisp
subjective. BTW Lisp ticks the other boxes. Personal focus on “maintainable”.
To merge two lists: use
append.Here’s the code formatted with more indentation (the right way):
Did you learn the
condmacro? Minimal knowledge is required to read Lisp as with any other language.Sure, but what I posted is (pretty obviously) teaching code. The function is called
myappendbecause (presumably) the teacher has told students, “Here is how we might writeappendif it didn’t exist.”Thank you, but that did nothing to make the code more readable to me. (See below on “to me.”)
I’m not sure, but you seem to be using “subjective” as a way of saying “wrong.” Or, to put this in another way, you seem to want to correct my (and Carl’s) subjective views. As a general rule, I don’t recommend that, but it’s your choice.
I’m glad you enjoy Lisp and find it productive. But you may have missed my point. My point was that I—and many people—do not find List readable or maintainable. (It’s hard to maintain what you find unpleasant and difficult to read, after all.) I wasn’t saying you or anyone else should change your subjective view; I was just stating mine. To quote jyx, “Yes, feeling is subjective and is fine.” I didn’t mean to pick on something you love. I was questioning how widely Lisp can play the role of great REPL plus readable maintainable programming language.
hey, right, I now find my wording a bit rigid. My comment was more for other readers. We read a lot of lisp FUD, so sometimes I try to show that the Lisp world is… normal, once you know a few rules (which some busy people expect to know when they know a C-like language).
rewording: “dear newcomer, be aware that Lisp also has a built-in for this”. I really mean to say it, because too often we see weird, teaching code, that does basic things. Before, these examples always bugged me. Now I give hints to my past self.
OK, no pb! However I want to encourage newcomers to learn and practice a little before judging or dismissing the language. For me too it was weird to see lisp code at the beginning. But with a little practice the syntax goes away. It’s only syntax, there is so much more to judge a language. I wish we talked less about parens, but this holds for any other language when we stop at the superficial syntax.
but truly, despite one’s tastes, Lisp is maintainable! The language and the ecosystem are stable, some language features and tooling explicitly help, etc.
The static typing idea is something that “seems obvious”, but will fall down at the first implementation / deployment.
Shell is about composing tools at the operating system level, and even C and C++ are not statically typed at the OS level!
Linux distributions use ABI versioning, not API versioning, e.g.
https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
They must do this because they deploy different components at different times. When you do
apt-get upgrade, components are selectively replaced without recompilation. (Compilation is the thing that enforces static types!)Windows does the same thing. When you do a Windows update, it replaces DLLs. COM and whatever the modern equivalent is are dynamic, not static. You query for interfaces at runtime.
So static typing doesn’t scale even to a single machine – on either Linux or Windows – let alone multiple machines, which is where I use shell a lot. (OS X or Windows talking to Linux is another very common use of shell!)
This is one of the main points in my post A Sketch of the Biggest Idea in Software Architecture
Another good recent post about it: https://faultlore.com/blah/c-isnt-a-language/
So shell has the same issue as C/C++ – components can be replaced independently without global versioning, and you need to evolve in a compatible way. (BTW there used to be this thing called Linux FHS, which would solve some analogous issues for shell as ABIs do for C++ – https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard – however it doesn’t seem to have reached its goals)
So basically, if you have a statically typed shell, you could only use it with a Google-like monorepo with a global version number and static linking.
Though even Google can’t redeploy every binary in a service at once, which is why protobufs have optional fields queried at runtime, not compile time. e.g. see Maybe Not by Hickey: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey)
So Google would still need a dynamically typed shell. Also, feature flags are the idiomatic way to upgrade services there as well. Yes, literal command line flags to server binaries, even though they have a monorepo and protobuf definitions.
https://featureflags.io/feature-flags/
So shells aren’t statically typed for the same reason that the Internet isn’t statically typed: you can’t upgrade everything at once. You don’t download the latest version of TCP/IP and link it into your programs like you do a gRPC definition. Instead you use the protocols dynamically (binary in the case of TCP/IP; text in the case of HTTP, HTML, JSON, etc.)
OP is talking about static types in the language level, and you seem to be talking about static vs dynamic linking, which … I think are kind of almost unrelated?
A shell is intrinsically a dynamic linker, just not in the sense that the term is routinely used. The purpose of a shell is to run other programs. Those other programs have interfaces (command-line flags, supported input and output) that change over time. You cannot map them into a static type system without also baking in the supported version. New tools are installed more frequently than a shell is upgraded and add to the space of shell types.
The .NET team had some interesting ideas about adding a section to binaries that described the tool’s interfaces and allowed them to be surfaced in PowerShell, but even then a static type system can express only the tools installed when the program is written.
Well outlined! Classical example of an hypothetical mystical epiphany vs reality. For 16 years now, every time discussions about shells come up, there goes the compulsory reference on how PowerShell is one step ahead and how going that path would materialize the utopia.
Yes exactly, thanks
https://www.unison-lang.org/ tries to scale static typing to internet level. They avoid compatibility problems using novel method: code (and types) stored in binary form in content addressable storage (I think this is Merkle tree). If old code (and types) are used, they are not deleted and are kept around forever. (Unless you stop to use them, then they are AFAIK garbagge-collected.)
Unison definitely has some cool ideas, and I’m glad someone is trying it
But it’s very far from how computers actually work today, and ironically their theory is to NOT have a shell at all, whether statically typed or not.
They’ll certainly have to solve open computer science problems in order to scale to even what a single Windows or Linux machine has today – e.g. ~30M to 100M lines of code. Let alone writing something like Google Maps, Twitter, etc. I’d guess they’re 100x to 1000x slower right now, or maybe exponentially slower, though I’d be interested in some real data
Nobody’s mentioned PowerShell yet? For shame. It has exactly that sort of “expose code API as shell options” behavior.
It was mentioned. https://lobste.rs/s/sqtnxf/shells_are_two_things#c_b3mny5
Powershell does a much better job of being both “terse and readable” with several levels of syntax.
Those footnotes just hit different
Very good points but I think general-purpose programming languages are simply not only not optimized for terseness, they’re usually incapable of being terse-but-still-descriptive.
My shell scripts are most often just wrappers that fix paths or provide default args or chain some calls. If a language (which is not Scala or Haskell) could be made to behave like that, e.g. by doing something like this:
but the margins are razor thin. It all breaks down if I have to do
and so on..
I’ve been banging this drum for years, I’m happy to add another great post to the pile of examples for when people ask me “why do you hate shell scripts”!
Another point to add is that shell scripts, or any scripts really, are fairly personal. Not that all programming is somewhat personal, but scripts especially. So seeing an entire OS propped up by these incomprehensible-to-all-but-one stringly-typed soups of seemingly arbitrary characters would make me feel uneasy too if that was my daily-driver.
And of course on the flip-side, one could argue it’s better to see the soup than not be sure if it even is soup behind those closed source .dll files on my system… but ignorance is bliss!
I’m perfectly happy with fish for (1), interactive use. But I want a better shell for scripting. I’m hoping Oil will succeed there. At work I write lots of one-off scrips to soft transition codebases. The script might evolve into a complicated 10 step process that automates as much as possible while requiring input from me at certain steps. Usually I end up combining many different tools: awk, rg, fastmod, python, etc. Some parts might need to be parallelized and use the filesystem for mutual exclusion / synchronization. This kind of thing would be so much easier with a shell that has actual data structures.
The proposed solution of an API with a thinly wrapped auto-generated CLI is not terrible. I have heard it is common within Google, for example.
In the Python world, there are various solutions starting from plac/argh and moving on to Click/autocommand and probably N others.
Personally, I prefer Nim to Python which has cligen. As mentioned here, but not in the article, the overhead of dispatch to a program in shell REPLs can also be thousands to millions of times higher.
A statically linked lua without readline is smaller than most shells. I wish more systems would use it as their default scripting environment, rather than writing shell (or, worse, bash) scripts for complex tasks. Lua isn’t the best language in the world, but it’s far from the worst.
Lua plus dedicated additions to the standard library (e.g., expanded filesystem operations and string support) would be heaven as a replacement for scripting in shell.
I recently started using Lua for scripting a thing in a videogame. I find it so weird to be using a language which has advanced features like coroutines but also doesn’t have an equivalent of string.slice() in the standard library.
Is not
string.sub()a suitable replacement forstring.slice()?Yes. Fuck! I missed it because I incorrectly assumed it was a variation on gsub. Damn.
Thank you.
The standard library is very small. I wrote a
splitmodule for Lua, since that’s something that I find essential which isn’t part of Lua’s standard library. I think that some hosts expand Lua’s standard library to help out (and to suit the needs of their users).In fairness, Lua is designed to be used as an embedded language. But I wish there were an optional “batteries” package. That said, LuaRocks is often helpful.
you should check out Hilbish then :)
https://rosettea.github.io/Hilbish/
Maybe the solution is to go the other way? Some sort of OpenAPI like spec that allows you to define an API for a command - but the result is still a command line call, and the spec doesn’t have to be attached to the command but can be shipped with the package in a designated folder/format. That way, the people writing the shell can do the work of integrating commands that don’t support this format.
Have you seen powershell? It sounds a lot like what you describe. I mean, they have .NET reflection instead of “OpenAPI”, but I believe it implements what you describe.
There are several problems with the reverse direction.
string. You may get number/int/float granularity, but, it’s rough. Meanwhile, prog.langs can often both print&parse all kinds of rich types like enums, lists, sets, tables, ..gccdiffers fromGNU ls.Your OpenAPI/spec idea could help with 1), but it would be hard to auto-infer from
--help-like docs. That means a lot of work on API wrapper writers (much like work for “TAB completion systems”, only maybe more so). Another way to think of 2) is that the “forward direction” (generating a CLI for an API) lowers the syntactic entropy/chaos of the world (by using just 1 toolkit/CLI syntax) while the reverse direction must confront it.Anyway, it’s probably easier to just build/map “shell-like syntax” (IO capture, redirection, pipeline construction syntax, ..) into your prog.lang for “faster porting” of REPL-prototyped code, leaving the burden on ultimate end-users to synthesize their string list argv’s.
My impression from Gentoo, Nix etc. is that if you can get a few nerds interested in your system, the amount of effort they’re willing to put in to create compatibility can be much greater than the sum total of effort you can get upstream to commit to supporting you.
That’s why I think it’s more promising to keep the glue layer external.
(Though I also love the idea of in-language shell support.)
LOL. The Zsh completion people indeed put a lot of effort in! There may be other technical difficulties that maybe the .NET team published details about. I think a lot of it is just the natural language processing trying to go “from docs to parse types”. http://docopt.org/ is another effort along that arc which might have insights. There may also be an implied requirement that the target prog.lang have sum types, e.g. if the command can parse a parameter where “digits mean one thing and text another”.
I would like to see better libraries in Python and Go that make it super easy to run commands, connect their pipes to each other, check error values, read results, etc. Perhaps using shell-like syntax.
https://github.com/bitfield/script
While this doesn’t solve the problem I love using the Elvish shell. It’s not statically typed but it has enough programming language features such as lists, maps, and builtin functions that I can use it for the vast majority of my scripts.
the python part is broken: reverse/reversed work exactly like sort/sorted , and the euthor of the post used reverse where reversed shuold be used