1. 11

I’m thinking about what interactive completion features to add to OSH [1]. Should I emulate bash (complete/compgen/compopt builtins), or do something else?

Possible answers to the survey:

  • I use bash with distro X (e.g. Ubuntu) with the default completion scripts. No customization.
  • I use bash with distro X + my own completion scripts. (Links would be helpful. I use some pretty old crappy scripts I copied here [2]). Does distro X use the bash-completion project? [3]
  • I use zsh / fish / etc. with completion scripts …
  • any other custom setup is interesting

I know very little about how zsh or fish do things. I’ve looked at some zsh completion scripts, and they seem like a different brand of gobbledygook than bash, but gobbledygook nonetheless. (What I mean that is basically that string manipulation and array manipulation in shell is really awkward, and completion scripts do a lot of it)

I realized that one of the most important completion scripts I use is git’s – I need it to complete subcommands, branch names, and sometimes flags.

It is 3000 lines and constantly being updated! https://github.com/git/git/blob/master/contrib/completion/git-completion.bash

So I figure it might be easier for OSH just to emulate bash as a start. It will probably be less work to do that then to rewrite 3000 lines of bash completion in some other hypothetical nonexistent system.

However I’m open to any diverging views. Does fish have good git completion, and if so how do they do it? Maybe they don’t need to be super up-to-date because most people only use a small subset of git commands?


Issue 175: Look into what it takes to run git bash completion

[1] Although I’ve explained that Oil is more of a programming language / systems language project than an interactive shell, I still want to use it myself instead of bash. I think it is mostly there, except for completion, and probably basic history substitution.

http://www.oilshell.org/blog/2018/01/28.html#toc_2

[2] https://github.com/oilshell/oil/tree/master/devtools

[3] https://github.com/scop/bash-completion

https://packages.debian.org/source/sid/bash-completion

  1.  

  2. 8

    I use fish with absolutely no custom configuration of completion scripts. I haven’t actually looked up how to do this yet, even though I can think of a few situations where I don’t like how it tab-completes by default.

    1. 5

      I use bash on Debian, with whatever defaults it has.

      A crazy idea: when you get down to it, completion is really about defining a grammar for command-line options - “a command can be the token ls, followed by zero or more of -a, -r, …” where some of the grammar productions are defined dynamically (“filename” being the most obvious example). I’d love a completion system where I can dump grammar files in a standard format (PEG, yacc, EBNF, whatever) into a directory, and executables to produce dynamic productions into another directory; I feel like it would be a lot easier to write completion grammars in a declarative grammar syntax than the imperative-grammar-manipulation system that bash and zsh seem to use.

      1. 2

        It looks like this tool is grammar-based, or at least it’s a DSL and not imperative: https://github.com/mbrubeck/compleat

        I definitely think the imperative model is verbose. But I think you have the classic DSL problem in this domain too: need to be able to “escape” out of the DSL for special cases. @cmhamill, who mentioned compleat in this thread, said that it’s not entirely “flexible”, and I presume that’s what he means.

        1. 1

          That’s a pretty cool tool, thanks for pointing it out!

          It looks like the implemented grammar is a lot simpler than, say, yacc or OMeta, though. While DSLs do often need escape hatches, I’m not sure that the limits of this DSL imply that all command-line parsing DSLs are too limited.

      2. 4

        I use zsh which has the more comprehensive coverage with completions. Especially when you consider things like support for system commands on non-Linux systems. Note that zsh itself includes most of them and the separate zsh-completions project is only a small collection and of lower quality.

        Zsh’s is much the superior system but you’d have to emulate a whole lot more to support them. Completion matches can have descriptions which makes it vastly more useful. The process of matching what is on the command-line against the candidates is much more flexible and is not limited to dividing the command-line up by shell arguments - any arbitrary point can be the start and finish point for each completion candidate. And as that implies, what is to the right of the cursor can also be significant.

        My advice would be to take the zsh compadd builtin approach which is more flexible and extensible than compgen/complete, do your own implementation of _arguments (which covers 90% of most completions) and similarly your own _files etc. It’d then be straightforward for people to write completions targetting both oilshell and zsh.

        1. 2

          Hm interesting, yeah I am looking around the Completion/ dir in the zsh source now and it looks pretty rich and comprehensive.

          I also just tried out zsh and I didn’t realize it had all the descriptions, which is useful too. Don’t they get out of date though? I guess most commands don’t change that much?

          I recall recall skimming through parts of the zsh manual like a year ago, and from what I remember there are 2 different completion systems, and it seemed like there was a “froth” of bugs, or at least special cases.

          I will take another look, maybe that impression is wrong.

          I think the better strategy might be to get decent bash-like completion for OSH, and then convince someone to contribute ZSH emulation :)

          I guess I am mainly interested in the shell system that has the best existing corpus of completion scripts. Because I don’t want to boil the ocen and duplicate that logic in yet another system. zsh does seem like a good candidate for that. But I don’t understand yet how it works. Any pointers are appreciated.

          I’ll look into _arguments… it might cover 90% of cases, but it’s not clear what it would take to run 90% of completions scripts unmodified.

          1. 3

            The zsh descriptions do get out of date. The strings are copied by the completion script author, so if the --help text changes, the script will need to be updated too.

            Zsh’s completion system is vast and old, the best combination. That’s why the 2 engines exist still today, as there are a number of completion scripts that are in the old style. I believe that most of those underneath Completion/ are using the newer system.

            1. 2

              Of the 2 systems, the old, compctl system was deprecated 20 years ago. Everything under Completion/ uses the new system. I wouldn’t say there’s a “froth” of bugs - it is just that there is a lot to it.

              It isn’t the descriptions so much as the options themselves that can get out of date. The task of keeping them up-to-date is semi-automated based on sources such as --help output and they are mostly well maintained.

          2. 4

            I use bash on Ubuntu with no completion customization.

            1. 3

              This is the most pleasant thing I’ve used, by far: https://github.com/mbrubeck/compleat

              It’s not entirely flexible enough, but it is far more pleasant than anything else I’ve dealt with.

              1. 2

                Thanks this looks really interesting! But does it have a large body of existing completions? Are you using it, or is it something you’ve written a completion for.

                It looks more principled, but yeah I think the DSL approach falls down when you need to handle special cases, and there are a lot of them in shell.

                Example corner case: after -c in Python and shell, the remaining things are arguments, not flags. You don’t need to use -- to end the flags in that case.

                I would like something more principled than bash or zsh for sure. But I also don’t want to rewrite completions for the world, or even start an entirely new body of completions. I’m open to more info on this though.

                1. 2

                  So I only know of a a few pieces of software that actually ship with completions written for it (e.g., polyglot), but I do use it actively and write my own completions. It’s my go-to tool whenever something lacks completions and it starts to bug me.

                  While it can be somewhat limiting, it does allow you to do some reasonable interesting things since you can shell out. The only issue I’ve run into that I couldn’t work around was trying to write completions for the heroku command, which uses subcommands of the form heroku access:add, heroku config:get, etc. Something about the notion of a word boundary and the colons did not mix. I eventually abandoned the issue since Heroku started shipping decent shell completion with their CLI.

                  Some examples of the more elaborate ones I’ve written can be found here: https://gist.github.com/cmhamill/a7e39eb576f83292cfb09f2e3d0090ed

                  1. 2

                    Thanks for the examples! Yeah this does look nice.

                    If I were to do something like this, I’d probably want to let you mention an arbitrary function in the DSL, to solve the problem of needing to do “weird” stuff. I imagine it’s something like parser combinators.

                    Also I don’t think the DSL should hard-code its notion of tokenization, which seems like why you ran into : problem.

              2. 3

                I use Zim which sources it’s completions from zsh-completions[1]. It’s nice but I’ve not had to hack on it much.

                I also use fzf completions for searching .history (bound to CTRL-R) and for files with fd[3] (bound to CTRL-T).

                [1] https://github.com/zsh-users/zsh-completions [2] https://github.com/junegunn/fzf#fuzzy-completion-for-bash-and-zsh [3] https://github.com/sharkdp/fd

                1. 3

                  I use zsh and it’s suite of completion mostly. I would possibly use bash’s more often if various servers I administrate installed the completion package (which is usually broken out into a separate package), but they don’t and I don’t care to push that. (Again, happy with Zsh’s built-in library.)

                  Fish does a neat trick to supplement it’s completion – it parses the output of man! Pretty good for most commands, but some weird manpages will throw it for a loop. Those are far and few between IIRC, but I haven’t touched fish for a long while.

                  1. 3

                    I use bash on Debian with bash-completion package.

                    1. 2

                      Ditto. This means locally-installed applications don’t have any completion scripts (which is one way I notice I’ve typoed gti for git), but completion is pretty comprehensive for packaged applications, and it’s at most a minor annoyance to only have generic completion for non-packaged commands. I’ve never bothered trying to add or modify completion.

                      There are a handful of annoyances with it; one I note frequently is commands which only complete filenames with certain extensions (e.g. firefox will only complete .html files, not .png files or files with no extension).

                    2. 3

                      I use elvish, which has a great autocompletion UI for program names, and is probably capable of autocompleting flags too, I have not checked. What I like about it is that when you have written a command name, or part of a command, if you press up it’ll search in its history for lines that match what you have on the prompt, so if you write “git p” and press up it’ll probably match a “git push” or “git pull” you wrote earlier.

                      1. 3

                        I use bash on macOS, Debian and Ubuntu with bash-completion package. But actual I use the [CTRL]+[r] history search more often than I use autocomplete

                        1. 3

                          I use fish, and the completion has always been good enough that I’ve not spent any time figuring out how to change it.

                          1. 2

                            Bash on Arch, with bash completion and set completion-ignore-case on in my .inputrc

                            1. 2

                              I use zsh but I found oh-my-zsh to be too heavyweight for my liking. I use a bunch of zstyle completion options, and readymade completions for git, racket, pip, systemd, pacman, Rust and OCaml.

                              1. 2

                                I use OpenBSD’s ksh on OpenBSD, HardenedBSD (shells/oksh port) and Gentoo (app-shells/loksh). I wrote my own shell completions, basically something like: `set -A complete_doas – $(ls $(echo $PATH | tr : ’ ’) )

                                set -A complete_git_1 – clone pull

                                set -A complete_gpg_1 – –armor –change-passphrase –delete-keys –delete-secret-keys –edit-key –export –full-generate-key –import –list-keys –list-secret-keys –receive-keys –refresh-keys –search-keys –send-keys –sign-key –verify

                                set -A complete_ifconfig_1 – em0 lo0 wlan0

                                set -A complete_mpv – –ao –list-options –no-audio-display –no-vid –sub-file= –sub-font-size= –vo

                                set -A complete_pass – $PASS_LIST -c generate edit insert git push

                                set -A complete_pkg_1 – autoremove delete info install

                                set -A complete_pkg_2 – $PKG_LIST

                                set -A complete_poudriere_1 – bulk jail list ports status testport

                                set -A complete_vm_1 – console destroy install iso list reset start stop

                                set -A complete_sysctl – $(sysctl -aoN)

                                set -A complete_zfs_1 – create destroy get list set snapshot

                                set -A complete_zfs_2 – $PARAMS

                                set -A complete_zfs_3 – $DATASETS

                                set -A complete_zpool_1 – list status`

                                1. 2

                                  I just let Fish parse my man pages to get completions.

                                  1. 2

                                    I use Ubuntu with bash and default completions. I am somewhat interested in extending them, but I once tried to look into how to register a new completion, and got terrified by what I saw, so postponed the idea till who-knows-when for now. (Though I feel that this might become somewhat more approachable in future if I’ll manage to migrate my home directory to Nix+home-manager eventually.) edit: And by the way, I have set completion-ignore-case On in my ~/.inputrc.

                                    1. 2

                                      As a CLI developer I would really like to see a protocol being developed for language-agnostic completion. Similar to --help and --version being semi-standard flags that are supposed to exist in program, what if there was a --completion [args...] command that would output some data structure, where [args...] is the array of arguments currently being entered. While typing the command the shell would call the program and then use the result as a guide for completion. Something like https://langserver.org/ but for command-line.

                                      I know this is not likely to happen, just applying my wishful thinking :)

                                      1. 2

                                        I use bash with Ubuntu with the default completion scripts. No customization.

                                        1. 1

                                          I use bash on NixOS with whatever completion it provides by default (program names in PATH and filenames in current directory). I also use most shells from within Emacs shell-mode, which I think does its own completion.

                                          I get incredibly frustrated by certain completion systems, I think Debian’s default, which will sometimes refuse to tab-complete filenames unless they contain certain patterns. For example mplayer or vlc commands will only tab-complete filenames if they contain strings like .mp4 or .avi. I find such behaviour surprising and obnoxious, so would highly recommend not doing that in other tab-completion systems :)

                                          1. 2

                                            Yeah I get frustrated with them too. It’s not so much that they only complete .mp4 only by default, but if you want to change that, you are confronted with a nightmare of inscrutable and untestable code !!!

                                          2. 1

                                            On some machines I use zsh+zsh-completions & zsh-suggestions. On one machine I use bash with hstr, which is pretty nice.

                                            1. 1

                                              You should have a look at how git autocompletion is generated. From what I understand, either very soon or very recently it will be/was changed to generate the bash completion from the command line argument parsing code rather than being written from scratch.

                                              1. 1

                                                That sounds interesting – do you have a pointer? I have looked at the git completion, but it doesn’t look autogenerated:

                                                https://github.com/git/git/blob/master/contrib/completion/git-completion.bash

                                                https://github.com/oilshell/oil/issues/175

                                                1. 1

                                                  I think the googleable terms are ‘compgen’ and/or ‘gitcomp’.

                                                  1. 1

                                                    Hm neither of those turned up anything, and the combination didn’t either.

                                              2. 1

                                                I use zsh’s normal completion system, setup using the menu system that came up the first time I ran it.

                                                1. 1

                                                  I use fish on Manjaro with the default completion behavior.