Unrelated to the article itself (though a wonderful read), I really appreciate the amount of thought and effort that’s going into Oil and related documentation. It’s quite rare to see such concerned effort in understanding the problem domain from first principles and not presupposing anything about any solution.
The ideas and methodology here will surely become a touchstone for future projects of the same ilk.
I’m not sure if I’ve missed it or whether it’s in the pipeline, but a survey of interactive shell features might be interesting, especially around auto-complete, history-search-as-you-type, etc. Fish does incredibly well here, to the point where it feels like magic (“How does it know I want to run this command/script by just the first character?”), which probably comes down to a few pieces of metadata:
Which directory was the command run in?
Does the command point to a local file, and does the file still exist?
Which command was run before the one I’m completing now, and was it the same command as in this session?
How many times has the command run in total?
When was the command run last time?
I haven’t read the code for Fish to know for sure, but it feels like the history suggestions are prioritized based on at least a few of these attributes, and I’m not aware of any other shell that does it as well.
Although I have cut the interactive shell out of the project proper in favor of a stable IPC interface that other people can use to make interactive shells (including GUIs and non-terminal based shell interfaces). This is called “headless mode”. It exists but I need people to bang on the other side of it:
So basically you’re not gonna get an interactive shell based on Oil until someone else makes a big effort :) And I would say that Oil is probably the only POSIX/Bourne shell that is tackling this. (e.g. busybox ash or dash don’t seem to have much ambition in this area; I talked with one zsh dev and he hopes Oil succeeds :) zsh appears to have significant tech debt in this area.)
(As always, feel free to chat with me on Zulip, etc. – all links on the home page)
It’s a huge project so I have to cut some things out – unfortunately the interactive shell is one of them. But the good news is I think things will go faster if people build separate projects, rather than needing to familiarize themselves with Oil source code. (I don’t think this is hard because it’s all plain Python essentially, but some people have found it hard.)
It might be best to think of Oil as “a shell inside containers” or a “distributed shell”. That has always been the primary focus. I spent about a year on the interactive shell, and it wasn’t enough to attract contributors to that part, or users:
I still use it interactively when I release Oil, so it works, but there’s not a hugely compelling feature and I don’t have time to work on it. The Oil language is more important!
About fish: I wrote that “fish Oil is a good idea” last year:
Oh wow, that’s quite incredible, and headless mode seems like a much better ecosystem investment rather than some parallel effort that is hampered by historical baggage.
I’ll read through the links and see whether I can try my hand at some Emacs integration alaeshell, thanks for the comprehensive write-up!
It solves the “where does this command’s output end?” problem that scraping bash -i has. And the isatty() issue.
It is rough BUT one other person has written a client for the headless shell (Subhav). So it’s not just a random thing I came up with :) Please join us on https://oilshell.zulipchat.com/ if you have time for this!
When using globs, you also have to ask yourself what is the maximum number of files that will exist there. Unix programs have a limit on the number of arguments that they can take and can fail with E2BIG. I’m pretty sure xargs handles that out of the box.
Another lesser know trick of xargs is that it can be used to unquote strings like that:
I was the “user asked why you would use find | xargs rather than find -exec” and the article ignores the rest of that comment thread where we pointed out that find supports batching the pathnames in that command and is not necessarily slower than xargs.
While I don’t mind xargs, I think (a) some of the features extolled here are not part of standard xargs and (b) a lot of the objections to other approaches boil down to “I don’t like mini-languages that aren’t made of switches”
The mini-language issue is also important because people tend to generate those by concatenation from shell, which leads to injection bugs / attacks. A theme on the blog right now is “string hygiene”, which is a big problem with many shell scripts.
So it’s not just aesthetic, but relates to
Pedagogy: people do have a hard time learning more languages (that’s probably the top complaint with shell)
Security: mini-languages are often generated from shell scripts. If you can simply use shell, say via the $0 Dispatch Pattern, you can avoid the whole issue.
Except instead of xargs you want xargs -d ‘\n’ or xargs -d $’\n’ or the more portable (to e.g. the xargs you would find on openbsd) tr ‘\n’ ‘\0’ | xargs -0 because you definitely don’t want the default behavior when piping from find.
Unrelated to the article itself (though a wonderful read), I really appreciate the amount of thought and effort that’s going into Oil and related documentation. It’s quite rare to see such concerned effort in understanding the problem domain from first principles and not presupposing anything about any solution.
The ideas and methodology here will surely become a touchstone for future projects of the same ilk.
I’m not sure if I’ve missed it or whether it’s in the pipeline, but a survey of interactive shell features might be interesting, especially around auto-complete, history-search-as-you-type, etc. Fish does incredibly well here, to the point where it feels like magic (“How does it know I want to run this command/script by just the first character?”), which probably comes down to a few pieces of metadata:
I haven’t read the code for Fish to know for sure, but it feels like the history suggestions are prioritized based on at least a few of these attributes, and I’m not aware of any other shell that does it as well.
Thanks! Oil has a very strong foundation for an interactive shell:
Although I have cut the interactive shell out of the project proper in favor of a stable IPC interface that other people can use to make interactive shells (including GUIs and non-terminal based shell interfaces). This is called “headless mode”. It exists but I need people to bang on the other side of it:
http://www.oilshell.org/blog/2021/06/hotos-shell-panel.html#oils-headless-mode-should-be-useful-for-ui-research
So basically you’re not gonna get an interactive shell based on Oil until someone else makes a big effort :) And I would say that Oil is probably the only POSIX/Bourne shell that is tackling this. (e.g. busybox ash or dash don’t seem to have much ambition in this area; I talked with one zsh dev and he hopes Oil succeeds :) zsh appears to have significant tech debt in this area.)
(As always, feel free to chat with me on Zulip, etc. – all links on the home page)
It’s a huge project so I have to cut some things out – unfortunately the interactive shell is one of them. But the good news is I think things will go faster if people build separate projects, rather than needing to familiarize themselves with Oil source code. (I don’t think this is hard because it’s all plain Python essentially, but some people have found it hard.)
It might be best to think of Oil as “a shell inside containers” or a “distributed shell”. That has always been the primary focus. I spent about a year on the interactive shell, and it wasn’t enough to attract contributors to that part, or users:
I still use it interactively when I release Oil, so it works, but there’s not a hugely compelling feature and I don’t have time to work on it. The Oil language is more important!
About fish: I wrote that “fish Oil is a good idea” last year:
http://www.oilshell.org/blog/2020/02/recap.html#fish-oil-is-a-good-idea-link
Oh wow, that’s quite incredible, and headless mode seems like a much better ecosystem investment rather than some parallel effort that is hampered by historical baggage.
I’ll read through the links and see whether I can try my hand at some Emacs integration ala
eshell, thanks for the comprehensive write-up!Some kind of Emacs support would be amazing! In fact I mentioned that here:
https://lobste.rs/s/gbjp09/blurring_lines_between_shell_editor#c_elqwqo
It solves the “where does this command’s output end?” problem that scraping
bash -ihas. And theisatty()issue.It is rough BUT one other person has written a client for the headless shell (Subhav). So it’s not just a random thing I came up with :) Please join us on https://oilshell.zulipchat.com/ if you have time for this!
When using globs, you also have to ask yourself what is the maximum number of files that will exist there. Unix programs have a limit on the number of arguments that they can take and can fail with E2BIG. I’m pretty sure
xargshandles that out of the box.Another lesser know trick of
xargsis that it can be used to unquote strings like that:Quite frankly I’ve never needed to use xargs. For some of the examples here, using find with
-execis enough for me.It’s quite a good idea for a program, and I can see where it would be useful, but I’ve just never needed to use it personally.
Edit: I just realised this exact thing was touched on in the article itself. xargs can be faster than find -exec.
I was the “user asked why you would use find | xargs rather than find -exec” and the article ignores the rest of that comment thread where we pointed out that find supports batching the pathnames in that command and is not necessarily slower than xargs.
The article mentions find -exec + and mentions the compositionality of xargs.
Thanks. I figured there was more to it.
For the record, I’m still a die hard
find -execuser.While I don’t mind xargs, I think (a) some of the features extolled here are not part of standard xargs and (b) a lot of the objections to other approaches boil down to “I don’t like mini-languages that aren’t made of switches”
Part of it is the language cacophony, but one thing I will add to the post is that xargs is also more compositional.
It composes with pipes: you can do not just
find | xargs, butfind | grep | xargs, etc.https://news.ycombinator.com/item?id=28258695
This is very powerful and not a surface issue.
The mini-language issue is also important because people tend to generate those by concatenation from shell, which leads to injection bugs / attacks. A theme on the blog right now is “string hygiene”, which is a big problem with many shell scripts.
So it’s not just aesthetic, but relates to
Except instead of xargs you want xargs -d ‘\n’ or xargs -d $’\n’ or the more portable (to e.g. the xargs you would find on openbsd) tr ‘\n’ ‘\0’ | xargs -0 because you definitely don’t want the default behavior when piping from find.
Added here: http://www.oilshell.org/blog/2021/08/xargs.html#xargs-composes-with-other-tools
Along with some other updates
I’d never seen the bash syntax of
$'\n'!Does
"\n"work too?In the case of bash, you can just try these things. The answer is no.
Thanks!
I don’t have bash here … and AFAICT
xargs -dis a GNU-ism.Yeah, xargs -0 is also not POSIX but it’s a lot more portable than -d. In those cases you can just do tr ‘’ ‘\0’ | xargs -0 .