1. 7
  1.  

  2. 4

    Anyone who has written or attempted to write a completion script would recognize this as a feat of engineering. Congrats!

    1. 1

      Thanks! It’s not done yet but I’m now confident it can be done, after plowing through those initial hurdles. It was fun at first, but after several weeks it became a chore.

      I knew that completion was nasty, but it was even nastier than I expected. I should write this in another blog post, but I learned that the bash-completion project actually disregards some of the bash API and re-parses bash in bash!!! Argh.

      I need a break now and will return to completion later. If anyone wants to help let me know :) I think it would be a fun task for people who like to program in APL or brainfuck :)

      1. 2

        I learned that the bash-completion project actually disregards some of the bash API and re-parses bash in bash!!!

        That is… interesting. But not very surprising, considering that the complete -F/-C API is quite barebone: it just gives you a list of words, without any information about the syntactical structure.

        You asked about zsh’s completion system, but I am not sure whether you would really want to emulate it. Putting aside the complexity of zsh’s programmable completion API, you would also need to emulate the zsh language, which is probably only a “fun task” using your definition - and I think you are already having enough of those by emulating bash :)

        1. 1

          Yeah that’s a good point. I have seen some zsh constructs in the completion scripts that I’m totally unfamiliar with. And I learned there are 2 completion APIs in zsh – and old one and a new one – although I don’t know much more than that.

          So yeah it’s probably out of scope. But I got jealous of the the superior completions during my research :)

          On top of that, zsh also doesn’t reprint the prompt every time you hit TAB, which is starting to annoy me about bash, now that I know there’s a nicer way to do it! I’m relying on GNU readline, and I’m not sure it has a way to avoid that. I think zsh has its own non-reusable terminal code that does all this.

          1. 2

            And I learned there are 2 completion APIs in zsh – and old one and a new one – although I don’t know much more than that.

            Virtually all completion scripts nowadays use the “new” system. The old system was deprecated some 20 years ago, IIRC :)

            I think zsh has its own non-reusable terminal code that does all this.

            Yes, it’s called ZLE (Zsh Line Editor). GNU readline is pretty limited; if you are serious about interactive experience then at some point you will either want to make your own thing, or use a more advanced library. The most popular Python library these days is Python Prompt Toolkit, although it’s probably not super relevant as IIUC you are going to remove the Python runtime at some point?

            FWIW, Elvish (unsurprising surprise plug :) also implements its own terminal magic in the edit/tty package. Writing to the terminal is actually pretty easy, you need to know just a few VT100 sequences: moving the cursor, clearing an area (very useful for incremental update). Reading is considerably harder, because the sequences are much more complicated, and different terminals send different sequences.

            1. 1

              The main reason I don’t want to write my own is because I need vi bindings (set -o vi), and I think other people want emac bindings.

              I googled and found this: https://github.com/elves/elvish/issues/728

              If I weren’t a vi user I would have more flexibility :)

              I feel like the editing mode pretty closely coupled with the display issue? Or maybe readline has a hook where you can control how the completions are displayed below the prompt? I would like to reuse the keybindings but customize the display.

              I looked at Python Prompt Toolkit about a year ago. I remember it being too “big” to reuse, but it looks like it’s about 30K lines of pure code, and depends on the wcwidth package. So it might be possible. It’s really less about the code size and more what “dialect” of Python it uses. I don’t use decorators, multiple inheritance, etc. and “OPy” will never support that.

              Yeah I’m looking at the code right now, and it’s quite big and uses a lot of decorators. It’s unfortunate because the alternatives I looked at didn’t have vi bindings.

              Nice post here:

              http://ballingt.com/prompt-toolkit/

              This whole blog from the author of the bpython shell has a lot of great stuff about terminals. I checked out the elvish code and will look more if I decide to do anything here (although I think I just have to live with readline for the time being).

              1. 1

                The main reason I don’t want to write my own is because I need vi bindings (set -o vi), and I think other people want emac bindings.

                I may be over-ambituous, but with Elvish my aim is to eventually make the editor programmable enough that users can implement whatever kind of bindings they like. Anyway, using readline as a starter is always a safe choice and you can decide later which route you want to go :)

                This whole blog from the author of the bpython shell has a lot of great stuff about terminals.

                Thanks! I am reading through http://ballingt.com/blog/ now. FWIW, it is not accurate to say that ballingt is “the author” of bpython; he is one of its more prominent contributors.

            2. 2

              On a side note, I think that “boiling the ocean” and writing completion scripts is also doable. The man page and help messages of most commands are pretty regular; sure, they are not regular enough for one to write a generic parser for, but given a specific command (e.g. git), writing a parser for its dialect of manpage or help message sounds entirely feasible, especially so if you start from the troff sources of the manpage which has semantic markup. Moreover, for things like GNU utils the help messages are highly regular (they are generated programmatically) that you can virtually look at the source code and derive a CNF to parse it.

              After writing several (I suspect 3-6) such parsers, I imagine you should be to identify several parameters that fully describe the dialect, and you can write a parameterized parser to parse virtually every manpage or help message. You can even come up with some heuristics for identifying the parameters automatically (machine learning, anyone?).

              A side advantage is that you will also get updates for free. Bash, zsh and fish’s completion scripts all require manual updates when a new flag/subcommand/etc. is added.

              1. 1

                One thing I learned is that bash-completion already parses the help of ls and other GNU builtins! It does it dynamically, every time you hit tab.

                This is in contrast to zsh which tend to bake the logic into scripts (I think in a semi-automated fashion as well). This seems to cause a version skew issue, although I’m not sure how bad it is in practice.


                I would be interested in trying to come up with better zsh-style completions to share between Oil, Elvish, and other shells. I talked with someone about that here:

                https://news.ycombinator.com/item?id=18061851

                I think the general idea you outline might work – it just depends on how much work it is. Bash completion scripts are nasty, but the API is actually quite small! It’s just the compgen/complete/compopt builtins, plus some special variables like COMP_LINE and reading COMPREPLY.

                Also, a zsh dev replied here, and he actually said he contributed the bash emulation to zsh ! But it does have the problem I mentioned – bash completion is not up to par with zsh completion!

                https://www.reddit.com/r/oilshell/comments/9n7taq/running_bash_completion_scripts_with_osh/

                1. 1

                  Actually the ZSH dev makes the same argument I would have made – “declarative” doesn’t quite cut it. I think that’s what you mean by “CNF”.

                  https://www.reddit.com/r/oilshell/comments/9n7taq/running_bash_completion_scripts_with_osh/

                  Declarative will work for common commands. But the thing I really care about is complex commands like git, and that’s where it falls down IMO. The git team is CONSTANTLY updating this script:

                  https://github.com/git/git/blob/master/contrib/completion/git-completion.bash

                  It has 852 commits from 2006 to 2018. It is already somewhat “declarative” because they invoke git --listcmds rather than duplicating information. But I think the problem is just inherently difficult.

                  git is command I need the most help with! I can’t live without the prompt either, so we’re implementing the ugly (but thankfully simple) $PS1 language too.

                  1. 1

                    Replying to both of your comments here:

                    One thing I learned is that bash-completion already parses the help of ls and other GNU builtins! It does it dynamically, every time you hit tab.

                    That’s new to me. It is closer to what I think is the most promising approach, i.e. “bespoke parsers” for different commands.

                    I would be interested in trying to come up with better zsh-style completions to share between Oil, Elvish, and other shells.

                    Yes, yes, yes :)

                    Actually the ZSH dev makes the same argument I would have made – “declarative” doesn’t quite cut it. I think that’s what you mean by “CNF”.

                    Ah, I typed “CNF” because I jumbled “CFG” (context-free grammar) and “BNF” (Backus Normal Form)…

                    I am not sure about your use of “declarative”. What I have in mind is:

                    1. A parser reads the output of help message, and generates completion code in $SHELL’s language.

                    2. $SHELL runs generated completion code.

                    Maybe you are referring to the idea of writing a generic parser so that you only need to give it a few parameters? I do imagine that for complex commands like git some customization may be required and the generic parser may not be sufficient.

                    Also, if we are going to have a common intermediate format, I envision the completion pipeline to look like this:

                    1. A parser reads the output of help message, and generates this intermediate format.

                    2. A converter converts the intermediate format to $SHELL’s language.

                    3. $SHELL runs generated completion code.

                    Step 1 is what can be shared, and I expect the bulk of heavy lifting to live there. The converter in step 2 needs to be written for each different shell, but they only need to be written once.

                    Also, there is a distinction between what I am proposing, and how bash-completion and git’s completion reads help messages at runtime. I am proposing that we do this at build-time; this makes the pipeline much simpler. The downsides are

                    • You need to re-build every time a new version of the command comes out to pick up new flags

                    • If the completion script is newer than the command in user’s system, the user will see flags that are not actually supported.

                    However, I feel these are relatively minor downsides. Maybe we can remedy them by monkey-patching lists of valid flags at runtime, but that obviously leads to duplicate work.

                    1. 1

                      The git team is CONSTANTLY updating this script

                      I have also long suspected that such scripts are so long partially because the language (bash) is not expressive enough. I am now reading https://github.com/git/git/blob/master/contrib/completion/git-prompt.sh and trying to write an Elvish version with feature parity. Hopefully it will be < 1k lines of code :)