1. 14
  1.  

  2. 6

    Regarding completion as a specification of types, I’ve been trying to get people excited about shell-agnostic autocompletion. Basically a bunch of code or “declarations” for describing various command line tools.

    Dev Log #8: Shell Protocol Designs

    Shell autocompletion is pretty far from this at the moment. In all shells I know of, you can hit TAB and get a candidate that generates a SYNTAX error, never mind a type error.

    https://news.ycombinator.com/item?id=19277948

    That’s an interesting invariant that I’ve been thinking about: hitting TAB should never generate something that would be an invalid command (or more pedantically, generate a command prefix that would have a syntax error once the user fills in the rest of the command).

    You would THINK that shell tab completion already works that way. But it doesn’t in bash, and not even in fish or zsh.

    There is an good (but currently dormant) thread about completion on https://oilshell.zulipchat.com/ about this if anyone wants to discuss further.


    I think it’s possible to have some lightweight types in shell, especially with the recent “proof” that gradual typing works in practice (MyPy, TypeScript, Hack, etc.). I now have some experience with gradual typing via MyPy, and I like it:

    https://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/

    It might be better to prototype something in a clean-slate manner for such an experiment, but if anyone wants to try type EXISTING shell scripts (which is a good design exercise), Oil has a nice statically parsed representation for shell code that would be easy to type check on top of. No other Bourne compatible shells have that.

    There are still some problems with syntax in Bourne shells though, like:

    ls $x f<TAB>
    

    If $x is --color, you might want to complete auto, on, off (an enum). If $x is --color=auto or a filename or --, then you’d want to complete a filename.

    Hence I think gradual typing would be more appropriate, because you wouldn’t have to solve all of these problems to get some benefit. The type checking can skip such cases and still type check the rest of your code.


    tl;dr I think types in shell are a good idea eventually, but existing shells have more than one syntactic problem to overcome before they can think about types. I’d like to see what a clean slate experiment looks like though.

    1. 5

      I thought a lot about this. I’m writing a blog post about how such a shell would look like but I’m trying to condense a lot of ideas I’ve had over a few years and it’s taking a while, so I’ll just write part of it here.

      One of the main benefits other than obviously fixing “passed a string but needed a path” would be much better completion, so a much better interactive experience. I believe some shells can show a file selector, and with a distinction between file types you could only show those of the type you need, just like in a GUI file selector.

      The other thing I thought about would be that, by using static analysis or just a fitting abstraction, you could analyze any shell script and return a list of all files that would potentially be touched by it and the programs it calls, which would drag it about halfway to Shill.

      I’m not really worried about the type annotations or increase in verbosity; I’d wager most of it could be inferred and the rest could be autocompleted.

      1. 3

        You might want to check out this verified shell for a start on what might be done. @andyc had some interesting comparisons to his project on that one.

        1. 2

          Sadly that way lies PowerShell. It can maybe be done much better than that, but I’m skeptical mostly because if I want to do something that needs type-safety in a shell I’m leaning towards using a type-safe language as opposed to making my day-to-day shell require it.

          1. 3

            What’s wrong with powershell?

            1. 7

              Mostly that it feels like a bad compromise between bash and C#. e.g it’s got the bashism of “keep going when there’s an error” as default, but even if you switch it to “no, seriously, stop” you then have it’s exception handling, which doesn’t catch all exceptions, which is really confusing as the .Net work it’s based on would just fine…

              I like it’s approach of “pipes of objects”, but it also has all the fun of having long names for shell builtins, and the deeply irritating script signing just makes it feel like it’s a lab experiment that escaped, and not in the good way.

              What might work is making something with the usual design aesthetics of a typical unix shell, but having a type system around for “pipes of objects”, but it would need to make sure it’s still a usable shell at the end of things!

              1. 7

                FWIW I don’t agree with the original criticism – you could have types in a shell without making it like PowerShell.

                But here’s my impression of PowerShell and some feedback from others.

                Not only is it tightly coupled to the .NET VM, all the “processes” it runs appear to be too. It prefers to run “cmdlets” which are like .NET programs running in the same VM, and not external processes, from my understanding.

                Fundamentally Unix and Windows work in different ways, and PowerShell caters more to the design concepts of Windows. Windows is based on shared libraries and COM (binary protocols), and later .NET, while Unix is based on text.

                I’ve heard some good feedback about PowerShell, but a lot of bad too.

                https://news.ycombinator.com/item?id=12313179

                PowerShell is my guilty pleasure of the computing world. Once you’ve piped strongly-typed objects around between your shell commands, text scraping seems barbaric by comparison. The problem, of course, is that you can’t use all that power for much, since the pool of PS-compatible tools and utilities is much shallower than it is for Unix shells.

                Although I don’t think it makes sense to run a Windows shell on Unix, because you lose its advantages, it apparently does run there. And the VM issue apparently makes it very resource hungry:

                https://news.ycombinator.com/item?id=12313969

                • powershell, 14 pids(threads?), 3119M Virt size, 80160kb Resident.
                • bash, 1 pid, 22068kb Virt, 3976kb Resident.

                https://news.ycombinator.com/item?id=17466305

                I gave up on PowerShell when I found out that the return value of a function is the console output.

                https://news.ycombinator.com/item?id=12986621

                That’s nice. Can you pipe binary data without powershell corrupting it, or is that now a feature?

                Some harsh feedback from 2014:

                https://medium.com/@octskyward/the-woes-of-powershell-8737e5346b1

                PowerShell feels like it was built by people who had heard about command lines a long time ago and tried to recreate one based on stories passed down through generations of their ancestors.

                When I look at the syntax, I tend to agree. It’s like they the cargo-culted a bunch of bad design decisions like the -eq operator without understanding that syntax isn’t why people use shell. It’s just legacy. It’s basically the worst of both worlds: it’s not compatible, AND it has legacy!

                To me, the reason why people use shell is that it lets you integrate things that were NOT designed to be integrated. It’s glue which is vital to making real systems work. But if you couple everything to the .NET VM, then you’ve lost that property.

                1. 2

                  To me, the reason why people use shell is that it lets you integrate things that were NOT designed to be integrated. It’s glue which is vital to making real systems work. But if you couple everything to the .NET VM, then you’ve lost that property.

                  The irony is that Windows comes from OpenVMS heritage. VMS admins told me things such as data types and calling conventions were standardized to make cross-language development easier. That also makes integration by function calls easier. The .NET VM does something similar at the VM level. In theory, it could have a unique experience if a shell builds on that advantage. Instead, if your post is accurate, they threw away the best of VMS and Windows advantage to fail to emulate a UNIX advantage. Worst of both worlds.

              2. 2

                The question is whether you consider type-aware tab completion to be a useful feature. If so, then there’s an use case for type-awareness in shell.

              3. 1

                Why is this a concern/problem? The moment I start caring about types in my shell scripts, I full-stop and move to using something a bit more robust (e.g. python). I’m curious why creating an entirely new shell scripting language is ‘the solution’ to all the problems with current shell scripting languages when there are things which are better suited in existence right now.

                1. 2

                  I see this as primarily useful for interactive shells, because as you say, just use a different scripting language if you need one. In a typed shell, tab completion scripts would be obsolete, and nobody would need to ask how to rm a file called -f.

                  1. 1

                    That’s a good point I hadn’t considered.

                2. 1

                  I think there are a lot of ideas mixed up in here. Dynamic typing beyond “everything is a string” and commands that can operate sensibly on such types, commands that can communicate to tooling about what types they need for completion, static typing, and (at least I see it subtextually) the tension in needs between interactive shell use, simple scripts, and larger scripts that want to turn into more mature “programs”. Different projects have worked on different areas of this. For instance, Powershell fixes the stringly-typed problems (Powershell may have its own issues, but I think the idea of using objects other than just strings is an important step forward). I think one of the most interesting projects with static typing for shells is Caml-Shcaml. And (Shameless self-promotion) I have a project Rash that aims to solve the issues of script growth.

                  Ultimately I think the direction shells ought to go is to be swallowed into general-purpose programming languages and become extension DSLs. When you start talking about a shell using “real” data types, having “real” error handling, having components that can communicate reasonably, having “real” type guarantees, etc, you wander into the realm of needing a “real” programming language. Naturally I’m inclined to think that my project is in many ways the best incarnation of this so far, but there are many projects pushing this direction (with various different strengths), like Xonsh, Closh, eshell, a new one posted on Lobsters today called Janetsh, and several others.

                  By being embedded in a full, general-purpose programming language, a shell language can inherit all of the goodies of its host – libraries of functions, data structures, error handling, etc. But the embedded shell itself can stay a small DSL that focuses just on things like process and function pipelining, interactive syntax, etc. Granted, extra stuff for nice interactive sessions make this a much bigger project (eg. Rash does not yet have good completion, live syntax highlighting, etc), but it also provides a sane base (the presumably sane host language) to build these things and their extensions on. That’s a big step up compared to the insanity of bash and zsh customization which has to be done in their respective languages.

                  1. 1

                    Shouldn’t the title be “type-safety”? I don’t think “safeness” is actually a word…