1. 66

I think most readers on lobste.rs understand the Oil project by now. However, reddit and Hacker News audiences don’t yet understand it. If you’re confused by anything in this doc, or think a first-time reader would be confused, let me know!

  1.  

  2. 18

    Some feedback:

    I have seen many oil shell posts, but still don’t know what the heck the actual OIL language looks like.

    1. 4

      OK thanks, maybe I should link these at the very top of the “A Glimpse of Oil”:

      http://www.oilshell.org/blog/tags.html?tag=osh-to-oil#osh-to-oil

      They are linked somewhere in the middle, which is probably easy to miss.

      It’s sort of on purpose, since Oil isn’t implemented yet, as I mention in the intro. But I think those posts give a decent idea of what it looks like (let me know if you disagree).

      1. 7

        I’ve seen your posts and hat around and never really understood what Oil was really about, but this link is really wonderful. The comparison to shell, the simplifications, the 4 different languages of shell vs the two of Oil, it all really clicked. Really cool project.

        1. 3

          I agree with the others. Until I see what’s your vision for the language, I’m not motivated to get involved.

          The only example you give contains “if test -z $[which mke2fs]”, which can’t be what you’re aiming at.

          IMHO If you really want Oil to be easy to use, you should take as much syntax from Python or Javascript as you can. And use similar semantics too.

          1. 11

            I’m willing to be convinced that a new syntax would be better for shell programming.

            I’m not very confident that moving towards an existing non-shell scripting language will get us there.

            The problem I have with writing shell programs in some non-shell language is that I expect to keep using the same syntax on the command line as I do in scripts I save to disk, and non-shell languages don’t have the things that make that pleasant. For example, a non-shell language has a fixed list of “words” it knows about, and using anything not on that list is a syntax error. That’s great in Python, where such a word is almost certainly a spelling error, but in a shell, most words are program names and I don’t want my shell constantly groveling through every directory in my $PATH so it knows all my program names before I try to use them.

            I’ve also never seen a non-shell language of any type with piping and command substitution as elegant as bash and zsh, but I’m willing to be convinced. I’m afraid, though, anyone in the “Real Language” mindset would make constructions such as diff <(./prog1 -a -b) <(./prog1 -a -c) substantially more verbose, losing one of the main reasons we have powerful shells to begin with.

            1. 3

              Yes it has to be a hybrid. I talk a little about “command vs expression” mode in the post. I guess you’ll have to wait and see, but I’m aware of this and it’s very much a central design issue.

              Of course “bare words” behave in Oil just as they do in bash, e.g.

              echo hi
              ls /
              

              I will not make you type

              run(["echo", "hi"])
              

              :-)

              One of the reasons I reimplemented bash from scratch is to be aware of all the syntactic issues. Process substitution should continue to work. In fact I’ve been contemplating this “one line” rule / sublanguage – that is, essentially anything that is one line in shell will continue to work.

              Also, OSH and Oil will likely be composed, and OSH already implements the syntax you are familiar with. This is future work so I don’t want to promise anything specific, but I think it’s possible to get the best of both worlds – familiar syntax for interactive use and clean syntax for maintainable programs.

              1. 1

                For example, a non-shell language has a fixed list of “words” it knows about, and using anything not on that list is a syntax error. That’s great in Python, where such a word is almost certainly a spelling error, but in a shell, most words are program names and I don’t want my shell constantly groveling through every directory in my $PATH so it knows all my program names before I try to use them.

                tclsh is an interesting example of not having this problem.

                I’m afraid, though, anyone in the “Real Language” mindset would make constructions such as diff <(./prog1 -a -b) <(./prog1 -a -c) substantially more verbose, losing one of the main reasons we have powerful shells to begin with.

                You get constructs that look like this in pipey libraries for functional languages (the likes of fs2 or conduit), though they’re controversial.

                1. 1

                  Well put. There’s also loads of muscle memory built up that is hard to leave behind. That point keeps me off of fish; I like almost everything else about it, but I don’t see why it can’t have a separate bash-like syntax.

                2. 2

                  OK that’s fair. I’m on the fence about outside contributions – some people have contributed, but I think most projects have more users and more “bones” before getting major contributions. I’m really looking for people to test OSH on real shell scripts, not necessarily adopt it or contribute. (although if you can figure out the code, I applaud you and encourage your contributions :) )

                  As I mention in the post, the OSH language is implemented (it runs real shell scripts), but Oil isn’t.

                  There will be a different way to test if a string is empty, but for the auto-conversions, if you have [ -z foo ], it will become test -z foo. The auto-conversion is going to make your script RUN, not make it idiomatic.

                  As far as appearance, you can definitely think of Oil as a hybrid between shell and Python/JavaScript.

                  I can probably write up a cheatsheet for those curious. I haven’t really done so because it feels like promising something that’s not there. But since I’ve written so many blog posts, it might be worth showing something in the style of:

                  https://learnxinyminutes.com/docs/bash/

              2. 0

                Yes and I don’t think I’ll care about it until I do. It could look like APL for all we know.

              3. 8

                Thank you for both the Oil project and this post. This is definitely the explanation I will point people to.

                I haven’t adopted Oil yet myself, and probably won’t until at least 1.0. I’ve tried zsh, fish, and xonsh, and have nice things to say about them all… but so far I always keep setting my login shell back to bash on linux, because there are just too many other people’s scripts for me to deal with. The net semantic complexity of $NEAT_NEW_SHELL plus that of $CRANKY_OLD_SHELL is always greater than the latter alone, so I find myself stuck with bash despite its irritations. It’s apparently just another one of these insoluble collective action problems.

                The embrace, extend, (eventually) extinguish approach that source translation enables is the only one I can endorse for having a hope of success in such an entrenched, messy, decentralized context as the Unix diaspora. There’s an important lesson here, and I hope similar projects take note.

                1. 8

                  but so far I always keep setting my login shell back to bash on linux, because there are just too many other people’s scripts for me to deal with

                  What does this have to do with the shell that you run? I run fish and that is no obstacle to running programs written in any other language, including bash.

                  1. 6

                    It’s not just the shell I run, it’s the shell “all the things” expect. I can easily avoid editing C++ or ruby source (to pick a couple of random examples) but, in my job at least, I can’t avoid working with bash. I can’t replace it, and I need to actually understand how it works.

                    Of course, other people with other jobs, or those who have long since attained fluency in bash, may have better luck avoiding in their personal environment. I can’t, because I have to learn it, ugly corners and all. I’d be happy to stick with fish, it’s just not a realistic option for me right now. My observation is that, for my current needs, two shells are worse than one.

                    1. 3

                      I’ve used fish for years now. Whenever I need to run a bash script I just run bash script.sh. The smallest hurdle I have to deal with is the small mental effort I have to make translating bash commands to fish equivalents when copying bash one liners directly into the shell.

                      1. 3

                        I don’t understand what working with bash scripts has to do with the shell that you run, though. Just because you run Python programs doesn’t mean your shell has to be a Python reple, these things are separate. In the case you’re referring to it sounds like bash is just a programming language like Ruby or Python.

                    2. 2

                      Thanks, yes I didn’t explicitly say “embrace and extend”, since that has a pretty negative Microsoft connotation :)

                      But that’s the idea, and that’s how technology and software evolves. And that’s is how bash itself “won”! It implemented the features of every shell, including all the bells and whistles of the most popular shell at the time – AT&T ksh.

                      Software and in particular programming languages have a heavy lock-in / network effects. I mean look at C and C++. There’s STILL probably 100x more C and C++ written every single day than Go and Rust combined, not even counting the 4 decades of legacy!

                      It does seem to me that a lot of programmers don’t understand this. I suppose that this was imprinted on my consciousness soon after I got my first job, by reading Joel Spolsky’s blog:

                      https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost-the-api-war/

                      There are two opposing forces inside Microsoft, which I will refer to, somewhat tongue-in-cheek, as The Raymond Chen Camp and The MSDN Magazine Camp.

                      The most impressive things to read on Raymond’s weblog are the stories of the incredible efforts the Windows team has made over the years to support backwards compatibility:

                      This was not an unusual case. The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented

                      This is a good post, but there are others that talked about the importance of compatibility. Like the “never rewrite post” (although ironically I’m breaking that rule :) )

                      https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/

                      Another example of this that people may not understand is that Clang implemented GCC’s flags bug-for-bug! GCC has an enormous number of flags! The Linux kernel uses every corner of GCC, and I think even now Clang is still catching up.

                      Building the kernel with Clang : https://lwn.net/Articles/734071/

                    3. 3

                      I’ve been seeing more and more about Oil Shell, but only now, after having have read this I have to say that I have found actual interest in the project. Until now, you only had a pretty website, but ideas like these really intrigue me.

                      But seeing that you’ve engaged with many different shell implementations,have you taken a look a stuff like Plan 9’s rc or related tools like mk? While I personally haven’t engaged too much with either, I know there are people who certainly see these as superior, and I’m guessing that there has to be something to it?

                      1. 3

                        Thanks for the feedback!

                        I’ve read the rc and mk papers, and discussed them with a few people on /r/oilshell [1]. This comment has an analysis of mk. As expected, there are good things and bad things about it:

                        https://lobste.rs/s/azldgm/oil_comments_about_shell_awk_make#c_rat8kv

                        It’s impressive given its age, but I think it has been surpassed. Ditto for rc. Although rc and es certainly are cleaner and smaller than bash, bash can do everything they can. As I recall, both of them were under 10,000 lines of C code, and there’s only so much you can do in that space.

                        More links here: https://github.com/oilshell/oil/wiki/ExternalResources

                        [1] https://www.reddit.com/r/oilshell/

                      2. 3

                        I agree that trying to maintain POSIX comparability isn’t worth it, but I’m interesting in the author(s) decision to separate out an oil shell and language. They’re trying to make a complete alternative for the shell, that can also run classic shell code. The fish approach is that you can still rush your bash/sh script, just like you can a Python or Ruby script; and that fish is for your interactive shell.

                        This post doesn’t really talk about fish, the shell I’ve personally been using it since 2013. It’s really an amazing shell and has come a long way, and it’s designed primarily around usability features and command line highlighting. Tab completion can be derived from man pages, and the highlighting, reverse searching and directory stack navigation are all incredibly useful.

                        So I gave oil 0.3.0 a shot. A couple of interesting things: Ctrl+C actually breaks you out of the shell. Tab completion for directory navigation doesn’t seem to work, although I do like the status bar at the top with information a search time. Another bug, if a user doesn’t have access to a bin directory, I get an Unhandled exception while completing: [Errno 13] Permission denied: '/usr/games/bin .. didn’t realize my user wasn’t in the games group.

                        I realize it’s still really early, and shells are incredibly incredibly difficult to write. Early versions of fish would craft often enough I’d wonder about the security implications. I applaud the contributors to this project and do hope we see more viable shell alternatives. Although I do suggest authors spend some time with fish as it comes with a lot of good stuff right out of the box, whose concepts I’d like to see incorporated elsewhere.

                        1. 1

                          I’m not sure what you mean with respect to fish. As far as I know, fish doesn’t run sh or bash scripts itself. You can of course invoke bash from fish, and you can invoke bash from osh too.

                          The first and last FAQ implicitly address your question about fish. Fish is “friendly interactive shell”. OSH is more concerned about the language for now, but I believe that will lead to a good interactive shell later. See the answers for details.

                          Thanks for trying OSH. I made a note of the Ctrl-C issue here:

                          https://github.com/oilshell/oil/issues/36

                          Also filed:

                          https://github.com/oilshell/oil/issues/69

                          I have looked a bit at fish, and in particular I like how they parse man pages for completion. I will probably steal that Python script! But that won’t happen for awhile, since I’m focused on the language.

                          At the very least, as I mention, OSH/Oil needs real functions so you don’t need to mutate globals to write completions!

                        2. 3

                          “Because the shell is terrible” is a sufficient answer. I am very excited to see where Oil and osh are going. I’d love to see what a new language environment designed under the same basic constraints as the shell would end up if bash compatibility weren’t important. I keep trying e.g. xonsh or eshell and always switch back to fish because as much as I prefer elisp or python to shell, the impedance mismatches are just too great.

                          Part of that, of course, could be the ~30yrs of Unix shell experience that I have burdened myself with, and were I starting from scratch, I might be able to escape the sad hexadecimal chains of unstructured 7-bit text.

                          1. 1

                            Thanks! By the way, I love the phrase “tilter at path-dependent windmills” in your profile :)

                            Shell is an extreme example of path dependence. Really, we ended up with this??? There is 40+ year unbroken chain going back to the Thompson shell. I’m pretty sure that technologies of a similar age like Lisp and Fortran have seem multiple “overhauls” since then.

                            And I think of bash as a friend-but-enemy… on the one hand, it helps me get things done faster than any other tool. On the other hand, I want to kill it. :) I’ve worked with a lot of non-programmers doing programming, e.g. including technical artists and data scientists, and I feel I can’t recommend that they learn bash “with a straight face”. Even though it will solve many of their problems. They usually prefer Python even though it’s not the best tool for the job.

                          2. 2

                            I’m not real familiar with Oil.

                            I like what I read, so far. Looks very promising. I recently wanted to write a static site generator, something real barebones, and I decided not to write it in bash; instead, I chose to write most of it in CoffeeScript, with some bash one-liners here and there. Coming from my angle, I guess I have two questions: why not just compile down to .bash? Is the runtime itself lacking? And also, given that I tend to rewrite as much bash as I can into CS, why would living in a Oil shell be better than coffee> ?

                            I read the Replacing Shell Scripts with Python post, and I can’t say I disagree with the author. I guess I am just unconvinced by many arguments on the page. Maybe just focus on the positives and put the rest in a -hater’s handbook.

                            Oil can and should succeed bash. I hope it does. Every time I write a bash script at work, often in the name of getting things done, I wince a little bit. I know I’m that coworker writing that smelly glue.

                            1. 1

                              Good questions. Yes, I want to improve upon the bash runtime too. A lot of it is syntax, but there are also semantic problems.

                              Right now, there’s no reason to ditch CoffeeScript for Oil. But I’m very much targeting command line programs written in Python / Ruby / node.js, so eventually Oil could be a replacement for what you do with CoffeeScript. But that won’t happen for quite awhile.

                              It’s roughly accurate to say Oil is a hybrid of shell and Python.

                            2. 2

                              As someone who is just starting to dive deep into operating systems, especially Unix, I’m grateful for all the writing you’ve done about the Oil project.

                              Oil is taking shell seriously as a programming language, rather than treating it as a text-based UI that can be abused to write programs.

                              One question in response to this statement is at what point does the shell language become just another programming language with an operating system interface. This question seems especially important when the Oil shell language targets users who are writing hundreds of lines of shell script. If someone is writing an entire program in shell script, what is the advantage of using shell script over a programming language? You seem to anticipate this question by comparing the Oil shell language to Ruby and Python:

                              …Python and Ruby aren’t good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what’s really going on.

                              So maybe these are good reasons (not sure if they are or aren’t) why Ruby and Python scripts aren’t clearly better than shell scripts. You also provide a mix of reasons why shell is better than Perl. For example: “Perl has been around for more than 30 years, and hasn’t replaced shell. It hasn’t replaced sed and awk either.”.

                              But again, it doesn’t seem to clearly answer why the domain language for manually interacting with the operating system should be the same language used to write complex scripts that interact with the operating system. Making a language that is capable of both should provide a clear advantage to the user. But it’s not clear that there is an advantage. Why wouldn’t it be better to provide two languages: one that is optimized for simple use cases and another that is optimized for complex use cases? And why wouldn’t the language for complex use cases be C or Rust?

                              1. 3

                                My view is that the most important division between a shell language and a programming language is what each is optimized for in terms of syntax (and semantics). A shell language is optimized for running external programs, while a programming language is generally optimized for evaluating expressions. This leads directly to a number of things, like what unquoted words mean in the most straightforward context; in a fluid programming language, you want them to stand for variables, while in a shell language they’re string arguments to programs.

                                With sufficient work you could probably come up with a language that made these decisions on a contextual basis (so that ‘a = …’ triggered expression context, while ‘a b c d’ triggered program context or something like that), but existing programming languages aren’t structured that way and there are still somewhat thorny issues (for example, how you handle if).

                                Shell languages tend to wind up closely related to shells (if not the same) because shells are also obviously focused on running external programs over evaluating expressions. And IMHO shells grow language features partly because people wind up wanting to do more complex things both interactively and in their dotfiles.

                                (In this model Perl is mostly a programming language, not a shell language.)

                                1. 1

                                  Thanks, glad you like the blog.

                                  So maybe these are good reasons (not sure if they are or aren’t) why Ruby and Python scripts aren’t clearly better than shell scripts.

                                  Well, if you know Python, I would suggest reading the linked article about replacing shell with Python and see if you come to the same conclusion. I think among people who know both bash and Python (not just Python), the idea that bash is better for scripting the OS is universal. Consider that every Linux distro uses a ton of shell/bash, and not much Python (below a certain level of the package dependency graph).

                                  The main issue is that people don’t want to learn bash, which I don’t blame them for. I don’t want to learn (any more) Perl, because Python does everything that Perl does, and Perl looks ugly. However, Python doesn’t do everything that bash does.

                                  But again, it doesn’t seem to clearly answer why the domain language for manually interacting with the operating system should be the same language used to write complex scripts that interact with the operating system.

                                  There’s an easy answer to that: because bash is already both languages, and OSH / Oil aim to replace bash.

                                  Also, the idea of a REPL is old and not limited to shell. It’s nice to build your programs from snippets that you’ve already tested. Moving them to another language wouldn’t really make sense.