@andyc, I don’t know if it’s encouraging for you or not, but the longer this project goes, the more excited I am to use it. Right now I think it’s still a bit “raw” for me, but I’m really looking forward to using it.
Thanks, yeah it’s not super surprising to me that it’s raw for end users. It’s sort of like if you build a C compiler from scratch – compiling libc is the bare minimum that somebody needs to do anything with it, but being able to so is a huge amount of work over writing a basic compiler!
Distros do a significant amount of work to make shells “usable”, including adding autocompletion, and that has to be done for OSH too. This isn’t hard because it’s very bash compatible, but it’s still work that someone has to do.
That said, it’s easy to test it out on real shell scripts now, and such tests often shake out good bugs. Obligatory “help wanted” :)
I’m still looking for a “killer feature” and I think that could be tracing or debugging (better than sh -x). However I think that naturally comes after speeding up the interpreter.
I think the interactive use will take a bit longer, since it has to bubble up through distro authors as mentioned, and eventually people writing tens of thousands of lines of shell code like oh-my-zsh, powerlevel10k, etc. In other words it takes a lot of people to make something that people can just use and mostly forget about!
Of course, the other “killer feature” was the Oil language, which exists and you can try it (in the slow implementation). However I think it will take many months to translate to C++, whereas OSH is within reach in 2020. So I’m looking for a different killer feature besides the Oil language to do first :)
I think using it continuous builds may be a good one, since they are often a mix of shell and YAML that is very difficult to debug. I wrote an alternative job runner in shell for Oil that works very well:
Very cool, I went to try this out. osh 0.8.pre6 compiled easily.
Seems osh’s read builtin is not yet up to feature parity with bash (4+? not sure); I tried running my own icd-bash and of read -rsN1, osh doesn’t like -s or -N (it says it doesn’t accept those flags). However, posix icd works well.
In a similar vein there’s also fff which osh dies on:
$ osh ./fff
((BASH_VERSINFO[0] > 3)) &&
^~
./fff:1102: fatal: Expected array or assoc in index expression, got value.Undef
Also, the most complicated bash (4.4+) program I’m currently aware of is bashtop. Granted, it’s not pure bash; it requires python3 psutil and GNU sed. @andyc, if you can get osh to run that, that would be extremely impressive.
BASH_VERSINFO is a common incompatibility, and I usually just edit it out. I think Crestwave has tested fff as well, but I don’t think it works entirely yet.
I just edited out version checks for bashtop, and then ran into the similar associative array / array usage issues.
I think bashtop could run just like mal Lisp does. The patches would be very similar and improve the program! If you or anyone has time for that I would help out on Zulip :) If the error messages are confusing or there are interpreter bugs I can fix them.
Yep, I did indeed see that osh was able to run neofetch (congratulations!). By complicated, I didn’t mean LoC… bashtop certainly does a lot more complicated drawing, at least. I’d assume neofetch is mostly a bunch of conditionals and platform-specific acrobatics.
Well rc is not POSIX or bash compatible, and Oil is, so that’s a huge difference. If you’re doing something in particular with rc that you’d like to do with Oil, I’m interested.
As I understand it, Oil as a program implements POSIX and bash languages, but also implements a new language that is meant to be better (which is basically the point of the project).
Rc is currently the main thing I see pointed to if you want to write shell scripts but have the language be more sane. So I guess I’m wondering about comparison of Rc the language to the “new” Oil language.
I’m still quite concerned that some unchecked over engineering is going on. There may be good reasons other shells didn’t need to be translated from python to c++ to run at efficient speeds.
But HTML5 parsers were translated from Python to C++. The main challenge was to specify it, not to implement it. I think HTML5 is the project most analogous to Oil.
Wow I didn’t know that HTML5 parsers were translated from Python! That is very related. Do you which browsers/engines did that? I couldn’t find anything after several queries like “html5 parser prototype python”
Another analogy I would make is that it’s a common goal for languages not to be defined by their implementation. Languages like PHP and Perl 5 are essentially defined by one implementation. But Rust has an effort to make a canonical version of the grammar:
For comparison, Go has a grammar in its spec, but it’s non-executable (and it’s also a simpler language syntactically).
So both OSH and Oil should be independently implementable, and the combination of DSLs re2c + ASDL + Python for OSH and pgen2 for Oil ensures that the languages don’t get tied to a particular implementation. Hand-written parsers written in C tend to grow a lot of unintentional hairs on them. (strings as buffers vs. strings as values)
@andyc Regardless, if you’re interested in performance, the first idea that comes to mind is it might be worth the added code to implement some popular and relatively simple external programs like cat and ls as shell builtins; fork + exec is very expensive, and many shell scripts I feel don’t make it a point to avoid this as much as possible.
Plus, in the case of ls, you’ve got Python to help out with the cross-platform bits, so most of the battle is already won. Although another major pain point here is that many rely on parsing the output format of these programs… then the BSD vs GNU vs POSIX implementations bite you. Multiple paths you could take here: implement all three and autodetect platform to branch, or standardize on “The Oil Shell ls” and make it a feature switch, introduce Oil-specific APIs to expose this kind of information, etc.
I never said it was. I just feel that if you had written it in C++ from the start, you could have skipped a lot of code that exists for no other reason. I guess saying that doesn’t help much, but its my honest opinion.
@andyc, I don’t know if it’s encouraging for you or not, but the longer this project goes, the more excited I am to use it. Right now I think it’s still a bit “raw” for me, but I’m really looking forward to using it.
Thanks, yeah it’s not super surprising to me that it’s raw for end users. It’s sort of like if you build a C compiler from scratch – compiling libc is the bare minimum that somebody needs to do anything with it, but being able to so is a huge amount of work over writing a basic compiler!
Distros do a significant amount of work to make shells “usable”, including adding autocompletion, and that has to be done for OSH too. This isn’t hard because it’s very bash compatible, but it’s still work that someone has to do.
That said, it’s easy to test it out on real shell scripts now, and such tests often shake out good bugs. Obligatory “help wanted” :)
http://www.oilshell.org/blog/2019/12/09.html#help-wanted
https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
I’m still looking for a “killer feature” and I think that could be tracing or debugging (better than
sh -x). However I think that naturally comes after speeding up the interpreter.I think the interactive use will take a bit longer, since it has to bubble up through distro authors as mentioned, and eventually people writing tens of thousands of lines of shell code like oh-my-zsh, powerlevel10k, etc. In other words it takes a lot of people to make something that people can just use and mostly forget about!
Of course, the other “killer feature” was the Oil language, which exists and you can try it (in the slow implementation). However I think it will take many months to translate to C++, whereas OSH is within reach in 2020. So I’m looking for a different killer feature besides the Oil language to do first :)
I think using it continuous builds may be a good one, since they are often a mix of shell and YAML that is very difficult to debug. I wrote an alternative job runner in shell for Oil that works very well:
http://travis-ci.oilshell.org/jobs/
Congratulations!
I’ve been following oilshell for quite some time now. Still use zsh, but I look forward to the day when I can leave it behind!
Very cool, I went to try this out. osh 0.8.pre6 compiled easily.
Seems osh’s
readbuiltin is not yet up to feature parity with bash (4+? not sure); I tried running my ownicd-bashand ofread -rsN1, osh doesn’t like-sor-N(it says it doesn’t accept those flags). However, posixicdworks well.In a similar vein there’s also
fffwhich osh dies on:Also, the most complicated bash (4.4+) program I’m currently aware of is
bashtop. Granted, it’s not pure bash; it requires python3 psutil and GNU sed. @andyc, if you can get osh to run that, that would be extremely impressive.Thanks for testing! I made note of the icd issue here:
https://github.com/oilshell/oil/issues/770
BASH_VERSINFO is a common incompatibility, and I usually just edit it out. I think Crestwave has tested fff as well, but I don’t think it works entirely yet.
I just edited out version checks for bashtop, and then ran into the similar associative array / array usage issues.
https://www.oilshell.org/release/0.8.pre6/doc/known-differences.html#values-are-tagged-with-types-not-cells
I think bashtop could run just like mal Lisp does. The patches would be very similar and improve the program! If you or anyone has time for that I would help out on Zulip :) If the error messages are confusing or there are interpreter bugs I can fix them.
bashtop does look quite substantial and I will add it here: https://github.com/oilshell/oil/wiki/The-Biggest-Shell-Programs-in-the-World
Although neofetch is more like twice as big at 10K lines and it runs under OSH :) After minor patches
Added an issue for bashtop: https://github.com/oilshell/oil/issues/776
more such issues: https://github.com/oilshell/oil/labels/should-run-this
Yep, I did indeed see that osh was able to run neofetch (congratulations!). By complicated, I didn’t mean LoC… bashtop certainly does a lot more complicated drawing, at least. I’d assume neofetch is mostly a bunch of conditionals and platform-specific acrobatics.
Is there a comparison somewhere of Oil vs Rc?
Well rc is not POSIX or bash compatible, and Oil is, so that’s a huge difference. If you’re doing something in particular with rc that you’d like to do with Oil, I’m interested.
As I understand it, Oil as a program implements POSIX and bash languages, but also implements a new language that is meant to be better (which is basically the point of the project).
Rc is currently the main thing I see pointed to if you want to write shell scripts but have the language be more sane. So I guess I’m wondering about comparison of Rc the language to the “new” Oil language.
I’m still quite concerned that some unchecked over engineering is going on. There may be good reasons other shells didn’t need to be translated from python to c++ to run at efficient speeds.
But HTML5 parsers were translated from Python to C++. The main challenge was to specify it, not to implement it. I think HTML5 is the project most analogous to Oil.
Wow I didn’t know that HTML5 parsers were translated from Python! That is very related. Do you which browsers/engines did that? I couldn’t find anything after several queries like “html5 parser prototype python”
Another analogy I would make is that it’s a common goal for languages not to be defined by their implementation. Languages like PHP and Perl 5 are essentially defined by one implementation. But Rust has an effort to make a canonical version of the grammar:
https://github.com/rust-lang/wg-grammar
For comparison, Go has a grammar in its spec, but it’s non-executable (and it’s also a simpler language syntactically).
So both OSH and Oil should be independently implementable, and the combination of DSLs re2c + ASDL + Python for OSH and pgen2 for Oil ensures that the languages don’t get tied to a particular implementation. Hand-written parsers written in C tend to grow a lot of unintentional hairs on them. (strings as buffers vs. strings as values)
Hmm, I can see how it might have helped with rapid prototyping.
Speed isn’t why it’s written in Python!
http://www.oilshell.org/blog/2019/06/17.html#why-is-it-written-in-python
@andyc Regardless, if you’re interested in performance, the first idea that comes to mind is it might be worth the added code to implement some popular and relatively simple external programs like
catandlsas shell builtins; fork + exec is very expensive, and many shell scripts I feel don’t make it a point to avoid this as much as possible.Plus, in the case of
ls, you’ve got Python to help out with the cross-platform bits, so most of the battle is already won. Although another major pain point here is that many rely on parsing the output format of these programs… then the BSD vs GNU vs POSIX implementations bite you. Multiple paths you could take here: implement all three and autodetect platform to branch, or standardize on “The Oil Shell ls” and make it a feature switch, introduce Oil-specific APIs to expose this kind of information, etc.I never said it was. I just feel that if you had written it in C++ from the start, you could have skipped a lot of code that exists for no other reason. I guess saying that doesn’t help much, but its my honest opinion.
Please read the FAQ I linked. It was started in C++ – second sentence.
Also read the bit about implementations not being specifications: https://lobste.rs/s/ct9kgg/oil_0_8_pre6_pure_bash_c#c_yekrzz