1. 19
  1. 3

    Maybe it’s not applicable at all but have you thought about choosing a leaner python implementation? Micropython comes to my mind. It’s targeted at embedded but maybe it’s a fit (?)

    1. 5

      MicroPython is an impressive project. I’ve looked at it a couple times and mentioned it on the blog.

      But it’s more than Oil needs and less than it needs at the same time… more because it implements a lot of the dynamism in Python that Oil doesn’t use (and which makes things slower), and less because it doesn’t have the same bindings (POSIX, readline). I looked at porting Oil to MicroPython, but it was too much work.

      It’s also more than 50K lines of code if I remember correctly, so to meet my goals of “one-person maintainability” I’d have to strip it down just like I did CPython. In particular MicroPython has an entire front end that I don’t need, because I have “OPy”.

    2. 3

      I really don’t understand the logic in implementing bash. It seems like 21 months of effort to re-implement bash and the real work of designing a better shell language isn’t started?

      Devil’s advocate, why is migrating old scripts to a hypothetical new language important? Why not have a way to compile a new shell into bash instead for deployment?

      1. 6

        Short answer: fish and zsh already exist, but they are generally used “on top of” bash. Linux distros and modern cloud systems all use bash:

        http://www.oilshell.org/blog/2018/01/28.html

        The goal is to replace bash, not just make a new shell.

        Another way to put it: a big motivation for Oil was digging around the guts of Debian and being horrified… :)


        Also, there’s a big list of other shells here: https://github.com/oilshell/oil/wiki/ExternalResources

        1. 2

          I fully agree with the mission, I still don’t understand why replacing bash involves reimplementing it? I suppose it is like a C++ situation? bash++ ?

          1. 5

            How else would you replace bash? Why didn’t any of the other shells I linked replace bash?

            Compatibility is really important – nobody has time to rewrite thousands of lines of bash scripts (and not just time, but expertise). If Oil isn’t compatible, they’ll keep using bash out of inertia.

            I myself have written thousands of lines of bash scripts, and I wouldn’t even port them into a new language wholesale! I would port them incrementally if there is enough benefit.

            The Python 2 -> 3 transition should give an idea how much inertia a programming language has, even to minor changes. That is, it was probably 10x more inertia than the Python dev team thought.

            1. 3

              To me the point is that it enables people with pre-existing bash scripts to move to a better shell system. This completely removes any barriers for people to migrate from bash to Oil.

          2. 3

            Although it wouldn’t lead to Oil, I considered new implementations for verifying correctness of configs and secure operation:

            Formally-Verified Interpreter for a Shell-like Language with interesting comments by AndyC

            SHILL: A Secure, Shell-Scripting Language

            1. 1

              Also, good question about compiling Oil to bash, answered here:

              https://www.reddit.com/r/ProgrammingLanguages/comments/9i3mnd/does_anybody_know_of_a_language_that_compiles_to/e6l6z80/

              It’s not out of the question, but I don’t think it’s as straightforward as it seems, or a clear win.

            2. 2

              Oil is a shell, for others like me who didn’t know: https://www.oilshell.org/blog/2018/01/28.html

              1. 1

                Why not use rpython instead?

                1. 2

                  In a similar vein, have you looked at python transpilers? For example, here is one that turns python into go then compiles it: https://github.com/google/grumpy

                  I suppose the output binary size would be fairly large though (Go binaries are pretty large in general)…

                  1. 1

                    The FAQ here addresses PyPy: http://www.oilshell.org/blog/2018/03/04.html#faq

                    I could probably add one on RPython specifically, but in short:

                    RPython is overkill because it handles the full generality of Python, which takes hundreds of thousands of lines of code.

                    That is, RPython and PyPy are bigger than CPython!!! The goal is to make something simpler than bash, and that would be more complex than bash.

                    And PyPy takes forever to compile with the RPython toolchain – more than 1 hour IIRC.

                    1. 1

                      Right but the question isn’t why don’t you use pypy - it’s why don’t you use rpython, which is something you don’t have to maintain or ship with the shell. It’s not clear to me why you want to write this on top of a stripped down interpreter rather than compiling.

                      1. 3

                        OK sure:

                        1. The build dependencies matter for a shell. Shells are often cross-compiled to embedded systems. For example every Android phone has a shell (mksh), which was ported to the somewhat quirky Android build process.

                        The RPython toolchain doesn’t fit there at all, and it supports many fewer CPU architectures than C or C++.

                        1. Interpreters written in RPython take forever to compile, so it will slow down the dev process. Compiling 10K lines of C++ code with no dependencies is super fast.

                        There is the “double interpretation” problem, and I will have an answer for that at some point, but it’s too early to get into details. There is a bunch of bootstrapping to do.

                        1. I eventually want to bootstrap Oil in Oil, so writing Oil in RPython is a distraction from that. This is pretty much like how the Go compiler was written in C, but it’s now largely written in Go.

                        RPython is very different from Python, and isn’t a pleasant language to program in (by all accounts). It would probably make the Oil code significantly larger and less readable.

                        1. JITing shell scripts isn’t desirable IMO. It’s a lot of extra complexity and probably won’t benefit most workloads (just like PyPy’s JIT made the OSH parser slower and use more memory, not faster.)
                  2. 1

                    So if I understand this correctly, OSH will end up being implemented in a custom subset of Python 2, assuming this “OVM2” will remain compatible with it. I can appreciate wanting to make a better version of Python 2 more geared towards the goals of your project, but I find the idea nonetheless devious, unless you’re planning to turn this OVM2 into the Oil language compiler at some point (therefore turning OSH into an Oil program)?

                    1. 3

                      Yes! The OVM2 work is not just infrastructure to support OSH. The Python-like semantics will be exposed to users as part of the Oil language.

                      Oil is basically a cross between bash and Python (with some features of Ruby (blocks) and JS). I said some of that further down in the thread I linked in this post:

                      https://lobste.rs/s/jrzwgy/what_is_systems_programming_really#c_xnvie9

                      This is of course a lot of work (writing a small Python interpreter), but the point of this post is to say that I’ve “audited” the situation and believe it’s possible.

                      1. 1

                        Nice, I’m looking forward to seeing how it develops.

                        By the way, I’ve noticed you mentioned removing all instances of closures from the OSH code; do you have any special plans for them in Oil?

                        1. 1

                          I guess I would want to sort out the difference between Python and JS closures. Are they the same? Python has the global and nonlocal keywords, but I believe that is mostly a syntactic difference not a semantic one.

                          I don’t see closures used very often in Python. The two usages I had in Oil weren’t “real” closures, as the captured variable didn’t outlive the parent scope.

                          Closures are a lot more common and idiomatic in JavaScript code. The combination of function literals and closures leads to a different and often elegant style in JS. Although this style doesn’t scale that well, and the JavaScript world seems to have moved onto other idioms (promises, virtual DOM to get your normal control flow back, etc.)

                          1. 1

                            I think they’re the same, I don’t know Python all that well but I don’t think there’s any difference.

                            Although this style doesn’t scale that well, and the JavaScript world seems to have moved onto other idioms

                            Trust me, closures are still used a lot in JS; I cannot see writing many parts of the React code I’m writing at work without closures. I think the reason why they’re not used in Python is mostly because Guido doesn’t like them, so he made them unpleasant to use.

                            Also, if you’re concerned about the captured variables in closures, have you considered making them explicit like in C++‘s lambda syntax? I’ve always thought it was a really neat feature which I would definitely steal if I ever wrote a language.

                            1. 1

                              Yeah I’d like to make sure JavaScript idioms with respect to closures work in Oil (although this is a LONG way off, probably at least a year.)

                              But yes a problem I have is that variable capture isn’t explicit! I usually prefer the explicit state of classes.

                              There are probably idioms I don’t use just because Python doesn’t really support them.

                              Another issue I had is that the difference between rebinding like captured = true and mutating like captured.field = true. Python supports the latter, but the former requires nonlocal. But I actually don’t see that many use cases for the former – it’s mostly the latter as far as I can tell!

                              In other words, I think Python is mostly missing function literals, which is a parsing thing, and it doesn’t actually have to do with nonlocal.

                              If you want to discuss further feel free to e-mail me at andy@oilshell.org, or https://oilshell.zulipchat.com/ . Although as mentioned keep in mind this is very far off :)

                              I would like to understand the implementation of closures better and how that affects their semantics. It’s not super straightforward as far as I can tell – the Lua 5.0 paper devotes a pretty large portion to it.

                              (I think the issue there is that the Lua compiler that generates code on the fly rather than generating an AST which you can do name analysis on. Shell has something of a problem like that because statements are parsed and executed one at a time. I was trying to get away from that for Oil though.)