1. 57
  1.  

  2. 14

    Congratulations on the release, so glad to see people working on new shells.

    In the past I have expressed skepticism of some of oil’s approach, but in the end he keeps pushing forward and I really think it is fantastic.

    Reimplementing bash is a giant task, andyc is a person of great stamina and hard work.

    1. 21

      Thanks! Yes I don’t blame anyone for being skeptical because the approach is definitely weird. I took an extreme approach to “make it work and then make it fast”. There are benefits to that, but also a bunch of downsides.

      And as I say in the blog post, it’s not a foregone conclusion that it’s going to work, even 3+ years later.

      But I do think that taking a weird approach is good if you have the appetite for it. Because you can’t expect different results taking the same approach.

      In particular my goal is to make something that’s “designed”, rather than the usual open source strategy of “design by accretion”. bash is of course the main example I’m thinking of, but there are others like:

      • git, in particular the UI. (It’s a high quality project in most other respects.)
      • C build systems, particularly autotools-based ones
      • Linux and its container design (namespaces, cgroups). We really paid for the lack of foundational work here, with projects like Docker on top, creating a lot of motion but little value.
      • JS and PHP (although, aside from their general sloppiness on corner cases, I think these languages got essential things right, which I’m taking cues from)

      So to put some design into it, I need some leverage over the size of the project, and I do that by implementing it in a high-level language. I concentrate on keeping the codebase small so I can tear it up at will, which I’ve done many times, and will do again in the near future.

      There’s also the practical matter than I know I can write Python code fast, and I know the internals of the Python interpreter so I won’t be “locked in” to a dependency.

      The approach has definitely caused me some grief, but it’s at least been an adventure. I’ve wondered if there was a more “MVP”-like strategy I should have taken. But I’ve realized it’s just not that kind of project. It’s more foundational, not really a “product”.

      The best analogy I think of is LLVM and Clang. Those projects are even more amazing, because they’re at least 1M lines of native code, and replacing/subsuming something that was around 1M lines (GCC). That’s 7-10x bigger than bash at 140K lines.

      And they’re still not done! Those projects started in 2000, but most Linux distros still use GCC, and the projects are still working on GCC compatibility (e.g. to compile Linux). But there were of course many other benefits to those projects (Rust, Julia, Clang’s tooling, etc.) so it was worth it, even though it took more than a decade.

      I do think if you don’t get the foundations right then you end up paying in complexity many times over again at the higher levels (looking at Docker again). So I think it will be worth it as long as Oil keeps making reasonable progress :)

      1. 6

        Congrats on the release, really looking forward to 1.0 :) Eggexes are pretty cool, but what I find ingenious is ability to gradually move away from bash to oil.

        As someone who started dipping their toes in containers, could you elaborate more on:

        Linux and its container design (namespaces, cgroups). We really paid for the lack of foundational work here, with projects like Docker on top, creating a lot of motion but little value.

        1. 7

          Thanks!

          As far as containers, the mechanisms in the Linux kernel have grown over time, without any sense of design.

          Here’s a pretty technical article about the cgroups API: https://lwn.net/Articles/679786/

          And then Docker went and put a bunch of stuff on top, without any sense of design either. A common complaint is that there doesn’t need to be a Docker daemon at ALL. It’s a moving part – a piece of code that has to be versioned itself – that doesn’t serve much purpose. You can just start a process and change its state in shell-like fashion. systemd has systemd-nspawn which does a lot of this (I haven’t used it, but I have used similar tools in the past).

          My opinion is probably biased because I worked with containers a lot pre-Docker. And I wrote some code against the Linux kernel APIs so I know how finicky are.

          I try not to be too negative about open source since I know everyone’s doing their best… but those areas in particular seem to be where a lot of problems bunch up.


          You will hear FreeBSD and Solaris users make fun of the lack of design in Linux too. I have never used those OSes to be honest, but they definitely thought about these problems before Linux did, and tried to address them in an integrated way.

          https://www.freebsd.org/doc/handbook/jails.html

          https://en.wikipedia.org/wiki/Solaris_Containers

          To be fair, they had “namespace isolation” before Linux. But Linux was the first to have “resource isolation” with cgroups. Hence its bad design.

          Basically Linux does well when it’s copying an existing design like Unix. When it has new ground to break, they generally do a bad job of designing the feature, because of the distributed development model.

          I guess things that are popular tend to turn into messes… the web is another great example of that :-/ But every once in awhile there needs to be some fresh design. We can’t just rely on the thinking that a bunch of people at Bell Labs did 40-50 years ago!

        2. 5

          BTW here are some more comments on Oil as a foundational project vs. adding more layers on top:

          https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj9/you_can_now_try_the_oil_language/

      2. 6

        Congratulations!

        1. 4

          Hm, regarding expression vs glob:

          However there are several keywords and sigils that put you in expression mode, e.g. so that * means multiplication rather than glob: (…) Inline calls also put you in expression mode:

          echo $strfunc(1 + 2*3) # Between (), you’re in expression mode

          Does that mean that functions arguments can’t be globs? Eg:

          do_something_with_files(data*.dat)?

          1. 6

            Good question, yes in expressions globs have to be quoted:

            Yes:

            ls *.py
            echo $myfunc('*.py')
            if (x ~ '*.py') {  # ~ operator also matches globs, not implemented yet
              echo yes
            }
            
            

            No:

            echo '*.py'  # not a glob
            echo $myfunc(*.py)  # syntax error
            

            So yeah you do have to have an awareness of what’s an expression and what’s a “word/command”, which is why I highlighted it.

            1. 3

              Eh, am I reading that right, that single quotes (the always safe but for literal embedded single quotes) being overloaded to turn globbing back on?

              echo $myfunc(’*.py’)

              Or is it just that ‘*.py’ will be passed unexpanded to myFunc and it might go though some hoops to glob expand the argument?

              How about a user specified myEcho - can it be used as “poor man’s ls” if called without braces?

              echo * # classic poor man’s ls in shell myEcho * # possible in oil?

              1. 5

                Right I should have clarified – they don’t turn globbing back on. It’s just a string. It’s up to the function that is called to interpret as a glob or not.

                It’s exactly like the difference between:

                from glob import glob; glob('*.py')  # yes glob
                os.listdir('*.py')  # no glob because it's not how listdir() works
                

                in Python. Single quoted strings in Oil are just like string literals in Python. Does that make sense?

                I’m not sure what you mean by myEcho? Both of these work in Oil just like they do in sh:

                echo *
                echo '*'
                

                Because you’ve never entered expression mode. You’re still in command mode.

                Here’s a short summary:

                • command mode: unquoted is a string, while $dollar is a variable.
                • expression mode: 'quoted' is a string, while unquoted is a variable.

                I started some docs that will hopefully clarify this more, but more questions are welcome and will help me write the docs.

                1. 2

                  Thank you for taking the time to reply.

                  So, if I’m not mistaken - a function in bash or ksh, will normally be called with globbing.

                  So eg:

                  myFunc *

                  some Built-in *

                  some-binary-in-path *

                  Will all be called with the rough equivalent of $(ls. ).

                  Does oil shell allow user functions to be called both like:

                  myfun args

                  myfun(args)

                  And do they have different (glob) expansion for “args”?

                  1. 4

                    Yes good question – this hasn’t been addressed by the docs yet, but it will be.

                    There are two kinds of composition / code units in Oil: proc and func.

                    • funcs are like Python or JavaScript functions. They accept and return typed data.
                    • procs are like shell “functions”. They look like an external process, with argv and an exit code. I think of proc as “procedure” or “proecss”.

                    procs are called with a “command line”:

                    myproc arg1 arg2 arg3
                    

                    funcs are called with Python/JS-like expressions:

                    var x = myfunc(42, 'foo')
                    do myfunc(42, 'foo')   # throw away the return value.
                    

                    This is NOT legal:

                    myfunc(42, 'foo')
                    

                    I will have a whole doc about this, along with some advice on where to use each. I do expect that it’s one of the more confusing things, but I think it’s justified because both mechanism are powerful and well-tested. I guess you kinda have to know shell AND Python to know when to use each.

                    I use shell as my “main”, if that makes sense. So generally speaking, procs calls funcs, and funcs won’t call procs as much.

                    1. 2

                      OK, I see.

                      How about stdin with “procs”? Can you pipe to (through) them? Ie, in bash/ksh(?) :

                      slow_cat() {
                        while read line
                        do
                          echo "${line}" 
                        done
                      } 
                      cat some_file | slow_cat
                      
                      1. 4

                        Yup absolutely, that was the gist of my post Pipelines Support Vectorized, Point-Free, and Imperative Style

                        There’s only one Oil interpreter, and a proc is executed the same way as a shell function. It really is a shell function, but it’s enhanced:

                        • You can optionally write a signature to bind $1 etc. to a name in a short fashion, and check that the right number of args was passed
                        • It uses a different keyword
                        • (not done yet) It’s a first class value, so you could do things like reflect on the docstring and other attributes, as you can in Python

                        This is why I made Oil a compatible shell – so I wouldn’t lose good features shell already has. There are some newer shells that invent worse versions of mechanisms that already work and are time-tested.

                        1. 4

                          Cool. Thank you for taking the time to reply.

                          I seem to recall seing some early notes on oil shell, and got a bit worried you’d gone completely off the rails ;-) Hapoy to see that does not seem to be the case :D

          2. 2

            Iḿ giving it a go and I can’t seem to set PS1. Or rather, I can set it, but it has no effect on the prompt. I tried with PROMPT_COMMAND=promptcmd too. Do I need to do anything more than set it?

            1. 2

              Thanks for trying it!

              PS1 is only for bin/osh at the moment. bin/oil is supposed to have something better, like a function hook, although it might be able to fall back on PS1.

              https://github.com/oilshell/oil/issues/498

              This is still an early release (as mentioned). You can use bin/osh now, and I’m looking for bug reports. But for oil I’m just soliciting feedback on the language (which is not stable, but has relatively stable syntax).

              1. 3

                That explains it. I read about how Oil is now a dialect within OSH, and didn’t realise that oil and osh are different things, when running oil presented me with a shell prompt. :) Now I’m trying osh and it runs my medium-complex bash stuff just fine, so now I can play around with both. Thanks!

                Also for people on NixOS, note that the oil package builds without readline; I submitted a pull request for that.