1. 19
  1. 9

    This is my attempt at “reverse psychology” and attracting contributors :)

    1. 2

      Also I posted an “easy translation task yesterday on Zulip: https://oilshell.zulipchat.com/#narrow/stream/325160-oil-discuss-public/topic/Easy.20translation.20task.3F (NO LOGIN REQUIRED anymore because Zulip just added “web public” streams.)

      tl;dr there is a test case that passes in Python but fails in C++ here:


      The error message tells exactly where it is (which is not always the case!)

      osh_eval.stripped: _build/cpp/consts.cc:1040: Str* consts::OptionName(option_asdl::option_t): Assertion `0' failed.
      osh_eval.stripped: mycpp/mylib.h:326: int Str::find(Str*, int): Assertion `0' failed.

      When I jump to the second location, line 326 of mylib.h, it’s simply a function we haven’t implemented yet:


      So we didn’t implement mystring.find('/', pos=42) in C++. We have the implementation for mystring.find('/') right above, so it should be very easy to fill this in.

      The hard part is really reading the test failures and navigating to the right part in the code. But after that it should be fairly pleasant – write a little C++ and then see the test turn green.

      (The first assert is a bug in a different Oil process! That confused me at first. That is also up for grabs though I didn’t look into it! If you want to become good at debugging (gdb/Clion) this is your project :) )

    2. 1

      I’ve noticed that many contributors try to work on OS X. I tried this at first too, but now I use a Linux VM under VirtualBox when I’m on OS X.

      A primary reason is that Spec Tests run against many shells, some of which aren’t built for OS X. A secondary reason is that it’s not clear what version of bash and Python are installed on OS X (although this is also an issue on Linux).

      @andyc maybe someone could tackle this by writing a shell.nix with pinned versions of python and bash? Even for non-OS X development this would be helpful for reproducibility. I’m not sure if this resolves the Spec Tests issue…

      1. 3

        Yes good question … So this has actually been tried by Travis Everett and a few other people, but there were some issues. I didn’t remember them at first, but your question jogged my memory and now I’ve recorded it on a wiki page:


        We do have a shell.nix, but I don’t believe anyone is using it now, and I think it has diverged from the shell scripts in 2 years:


        What did we learn from those Nix experiments?

        1. The spec tests aren’t isolated; they somewhat assume a file system that looks like Debian / Ubuntu / Linux. These can be fixed, but it’s laborious, and containers solved the problem much more cleanly.
        2. A more fundamental issue: the spec tests tickle EXTREMELY shell specific behavior including some from libc, which apparently Nix cannot sandbox on OS X. Although Nix can isolate many shared libraries, it doesn’t use containers – it uses a somewhat elaborate PATH configuration.

        But I believe libc is fundamentally a “hole” in the sandboxing of Nix, at least on OS X. (I’d be interested to be correct or learn more about how this works)

        This is one of those problems of “shell is lower level than you think”. It might be that Nix is a good solution for most projects besides a shell !!

        Off the of on my head, GNU libc glob() behaves differently than OS X glob() in rare corner cases like parsing character classes [a-z] and locales. There may have been some other libc corners that differed, like regexec().

        Now it’s a good question whether that’s a bug in Oil … I would like the shell to behave exactly the same on every machine, and not rely on libc versions. My assumption when I started this was that libc is so old that all these problems have been worked out. But apparently that’s not the case.

        Also, Nix does not solve the “it works on my machine” problem. Or at least there are ways to write shell.nix where you don’t have isolation.

        I learned about “Nix Flakes” at this time, and it seems like Nix Flakes was NOT ready yet, and it was what I THOUGHT that Nix was going to be. That is, I think containers have some better isolation properties than Nix.

        Though containers also have TOO much isolation. For development you want your vimrc and bashrc and so forth, and GUI tools. So this is a fundamental and hard problem.

        Anyway, so we needed to use Travis CI with Nix, because the isolation doesn’t ensure that things won’t break. But that support was experimental – it depends on a cache service called Cachix. (Later Travis CI got turned down.)

        So I just started using raw Travis CI on VMs, and I got the build green in a short amount of time. I used the Ubuntu image most similar to my own image.

        Later, I ported the CI to containers. So I believe contributors should use containers now. It is an easier solution than Nix, although it’s only barely ready. (I use it, but I don’t think anyone else does. They can get by with their system libraries.)


        So we want the same principle as Nix – the repository should have a configuration that specifies the development environment exactly. Whether that is a Nix expression or a container definition (Dockerfiles: https://github.com/oilshell/oil/tree/master/soil)

        Summary: in practice I was able to get our build green faster with containers. Otherwise, we would need to fix a bunch of spec tests, and even if we did, there is the fundamental libc on OS X issue.

        However there are problems with our container setup, some of which I’ve written about on the blog and Zulip:

        • It’s not really reproducible
        • I don’t want to depend on the Docker registry. In some sense Docker is “winning” because they provided a cloud service that is perhaps more ubiquitous than Cachix.

        I personally would like to move to a model where I can store containers on a plain old web server. I dislike the Docker registry API because it violates the Perlis-Thompson Principle!

        That was probably more of a response than you asked for :) But stuff like this is fundamental to the project, and I’m glad that we’re seeing the problems, so hopefully it can be folded into Oil and complementary projects in the future.

        So yeah I think this would be a good blog topic in the future.


        1. 1

          @ac experimented with creating a set of scripts exploring the solution space in vicinity of Nix and Docker, built on a much simpler set of primitives; I wonder if you’ve seen them and if they might be of interest to you? -> https://github.com/andrewchambers/p2pkgs/ (I have no idea if it works at all on OSX, though; I’d assume that most probably not).

          1. 1

            Yes I remember chatting with @ac about an earlier iteration on this a few years ago. It looks like it’s going in the right direction.

            I’d be curious to hear if IPFS is useful for it. (Most of the feedback I’ve seen has been bad, but I haven’t used it myself)

            Something like IPFS (or the more research-y Named Data Networking) is complementary to the “distributed shell” ideas I want to explore.

            So I’d say that Oil has a very concrete dev environment problem that’ s a great test case for any system like this. Writing the equivalent of “shell.nix” for Oil should shake out a lot of issues.

            I would have liked if Nix was the solution to the problem, but I think you can see we gave it a fair shake, and containers are more promising.

            Some future work here: https://github.com/oilshell/oil/labels/containers

            So if a new system can make that work obsolete, I would be very happy.

            I also agree with the peer-to-peer idea. As mentioned I would like to simply store code on a web server and not rely on centralized services like the Docker registry or even Cachix. I believe systems like this should “scale down” just like git does.

            (I know it’s “easy” to run your own Docker registry, but the CLI defaults to their clod, and I can’t get over the fact that it’s not just “files” …)

            1. 1

              As for Nix vs. Docker in case of Oil, personally I just don’t know. I’m happy you’re finding some success with Docker. As for Nix, I’m not an expert in it, never claimed to be, just an enthusiast who sometimes tries to use it here or there. At the same time, I think Nix requires much more of a change in thinking to understand and use it well - or, in simpler words, Nix is hard, and Docker is easier. And that’s a good thing for Docker for sure! Then, your way of structuring Oil’s build system and dependencies is IMO also rather nontraditional, at least for someone with my experience - and it is also great that it works so well for you!; yet for myself I found that it added an extra layer of complexity over Nix, which is already something that I’m still only playing with from time to time, and often get challenged by (though Docker too!). so as for me, I lost the energy at the crossing between those things. I know others also tried to help Oil work with Nix, maybe some of them more experienced, but still probably in some of their free time. I didn’t watch the situation, so I don’t know where they got. But then, if containers currently seem to work for you, I’d say that’s awesome, and go with them, why not! Yet I do also think maybe a day for Nix might still come some day, who knows. But also I don’t claim Nix is perfect or has no warts; it certainly has! I wish it e.g. had static typing, better docs, more uniform stdlib, better CLI interface, etc. But it’s also kinda a first of its kind, an experiment that worked, so it’s still plowing through the unspoiled fields of possibilities, evolving on the go and often carrying some history of local optima that showed up to be imperfect only in hindsight, far too late to change. So I do wish for a simpler, better successor at some point in the future. I also wish Mercurial’s sane interface won over Git’s accidental, and that some day a simpler successor to Rust will be developed. Yet for now, I’m happy and grateful for Rust, for Nix, for Git, as they are, and for Docker too! If always hungry for better ones. In a way, that’s one of the reasons I adore you for the Oil project, as as I see it you’re tackling a system (bash) that also evolved in an uncontrolled way to collect a lot of cruft and technical debt. But possibly also due to in some ways being innovative, or maybe even if not that much so, then for sure at least through the amazing virtue of being used and being useful. I can only hope some of the other systems might some day be lucky enough to interest someone like you!

              1. 1

                Thanks for the kind words, as always! Although I would caution against thinking my opinions are going to change much :)

                Oil is still what I’d call “default dead” … It’s kind of hard for me to reconcile …

                1. What I feel is very rapid progress on many fronts (e.g. I mentioned fixing 350 spec tests in C++ in a few days; making progress on the YAML-shell problem in a few days and only a few hundred lines of code!)
                2. The fact that there’s so much more left to do, and the fact that many subprojects are “buried” and you could argue “failed”

                For example Eggex is already almost 3 years old? People liked it here, and it got posted to Hacker News a few months ago, and that’s about it.

                The language isn’t alive unless lots of people are using it … and we’re really blocked on translation. There is very little chance I can finish it myself. I can easily see 4 more years of work for myself, bringing the project total to 10 years. So it needs to be parallelized …

                • headless shell seems to be dormant, and I think it’s a great idea. A contributor prompted me to flesh out that design, but he’s no longer working on it
                • I spent forever running bash autocompletion scripts with Oil… It works and I use it all the time. But I don’t think anyone else does. I sent queries upstream and nor response. I don’t blame them as these are things there is very little “staff” for

                So yeah I am trying to shake the project up as you can see, but I can definitely use help. In either the form of coding or testing or advocacy / recuitment. (Finding C++ engineers has been tough; I also want to make a call for Rust and other languages)

                You’ve probably have seen it, but this Build and CI post has some “philosophical” thoughts on the issue : https://www.oilshell.org/blog/2021/04/build-ci-comments.html

                I made several lobste.rs comments about it too – e.g. how using Bazel soured me on “opinionated” / pure functional approaches. (e.g. rewriting upstream problem)

                One slogan of the project is “Shell is the language of heterogeneity and diversity”. So I think shell should be used in opinionated tools like Nix, but also in many other places!

                I do think containers fit the philosophy of the project – they are a smooth upgrade from traditional Unix. Just like Oil is a smooth upgrade from shell.

                Nix is not a smooth upgrade at all! You have to buy in all at once. Even a blog post yesterday said that

                I think there is a HUGE opportunity to make containers actually compose, like a language … podman did some good things but didn’t go far enough. Unfortunately this is another one of those language design things that is buried under a lot of work :-/

                I might write a blog post about this … Kind of the reverse psychology / contrarian viewpoint that seemed to have resonance here :)

                1. 1

                  I’m curious what you mean by “blocked on translation”?

                  Personally, I feel I’d be more than happy to try using it once it’s usable as an interactive shell drop in replacement for bash; IIRC, last time I read it was not performant enough for daily use in this character yet, is that still so? I know it’s a huge ask probably, but just specifically for me personally in my situation, that’s what I think is the case at my current situation. This would allow me to hopefully use it daily as normal bash, and slowly be exploring the Oil features when having some moments of time.

                  On a semi-related note, I worked on a somewhat crazy and unreasonable (though much smaller than yours) project for a while (https://github.com/akavel/dali), where I was quite convinced there must happen some moment when I’ll run into a stumbling block that will be just too big for me to overcome. To my big surprise, this moment never happened and I got the project to the end. Although I lost some energy for it afterwards and didn’t push further work in it over the PoC, to make it shrink-wrapped and easy to use for people, and to advertise it strongly enough to make it break through. But I’m still happy I did it and think of it with absolute joy.

                  PS. As to eggex, I still think it’s very interesting. I used it in one of my projects just as a notation, although there was no eggex library for that language, it was just friendly to reason about. I certainly have it in my mind, and will try to use it if I find a good opportunity, where the fun<->cost balance is adequate. But it’s hard to predict when such a time may occur. But your seeds are sown in this area, and it is valuable.

                  1. 1

                    I feel I’d be more than happy to try using it once it’s usable as an interactive shell drop in replacement for bash; IIRC, last time I read it was not performant enough for daily use in this character yet, is that still so?

                    It definitely works as a drop-in replacement for bash now. The speed is slightly slower but not very noticeable, though in some cases it feels faster, probably due to fewer autocomplete plugins.

                    People have used it interactively for weeks or months and be fine … and maybe some people still use it?

                    But I would not say there is a great reason to use it interactively, which is why I don’t push it. Most people like something more zsh- or fish-like.

                    On the other hand, it is too slow to say run Python’s ./configure, and that’s why I’m working a lot on translation.

                    What I would say is that if the current trajectory continues, you’ll probably not have any reason to use it in 2 years either! Hence the “default dead” terminology.

                    I’m looking at translation now, and it’s a big task. There’s definitely interest, but my fantasy of having someone “take it over” almost certainly won’t happen.

                    Instead I will have to cobble together help from various sources and review it and integrate it. And likely apply for more grants and so forth. So that will take a lot of energy.

                    With luck, in 2 years we’ll have a fast pure C++ shell, and a bunch more Oil language features.

                    But I think by then you will STILL not have a great reason to use it as your interactive shell. For one thing, it will run fewer autocompletion scripts than bash.

                    That is one of those projects that I feel is buried, i.e. sort of a dead end in terms of growing the project. Over 3 years ago I spent tremendous effort to get that to work, and it’s barely moved since then. People are not building on top of that work or even testing it really. (I think this is partly because only very few distro and git maintainers understand bash autocompletion; most people don’t write their own)

                    I think I was fairly clear about that in posts like: http://www.oilshell.org/blog/2021/12/backlog-assess.html

                    So I am clearly trying to change the trajectory of the project … But I guess I’m pushing back a little to say you could be unreasonably excited :)

                    In one blog post I said that if you’re waiting around for a new shell, you will probably be disappointed. For me Oil the main purpose of Oil is a linguistic foundation for distributed computing (and I’m finally making big progress on the YAML alternative, which I’m very happy about).

                    But I’ve come to realize that this is essentially an open research problem, and building stuff on top of Oil is another 10 year project.

                    And people adopt platforms; they don’t adopt languages. (That is another one of my future blog posts)

                    So basically if you want to use it, I would think about ways you can change the trajectory:


                    Also I’m thinking of writing some “counterfactuals” to Oil:

                    • bash clearly works, and you use something like zsh on top. It’s suboptimal, but not enough for people to switch.
                    • YAML clearly works – there are more than 20 cloud providers that use it as an interface. Most people do not want a compositional language. They want to copy and paste YAML snippets.
                      • Having documented examples to copy from is what matters, not whether the language composes and lets you do creative things.
                      • YAML is worse than macro programming like m4 / autotools / C preprocessor, but that’s OK. Sometimes we just slide backward.
                    • Also Oil’s YAML alternative (Hay) is very much like UCL and HCL. Those already exist, but people use YAML. (The difference with Oil is that shell can be interleaved, and it has better programmability.)

                    This is a little bit like the “Bitcoin Maximalism” post by the Ethereum creator … basically trolling his own project, which I think is a good way of analyzing things.

                    1. 1

                      Hm; as to whether it is working as a drop-in replacement for bash for interactive use, I’m still confused now: it mostly is, but not completely, because some completion scripts I may have installed won’t work, is that so? Also, bigger script like various ./configure may work noticeably slower, is that so? (Though that surprises me, I would assume configure scripts spend most of their time running GCC and in I/O.)

                      As to pivoting Oil in a direction of YAML templating (i.e. Jinja etc), personally I’m not interested in this idea, if I understand it correctly. As you say, stuff like this has been tried more than once; part of the feature of config languages is that they try to be simpler than “full blown programming languages.” As such, personally the one thing I find interesting in this area is the CUE “language”. This feels like something that could actually maybe make understanding configs and avoiding errors in them easier.

                      One more thought that comes to my mind, if you’re looking for alternative strategies for Oil/OSH (hm; is part of my issue in this discussion that I’m constantly conflating those two?) as a bash-escape tool: I wonder if you considered the “commoditize your complement” trick? I.e. writing a translator from Oil/OSH to bash? Then, interested people (“Oil hackers/explorers”) could write their stuff in Oil and know that it will run safely both in Oil and in ye olde bash (after they processed it through the translator). Thus prefer to write scripts in Oil as it’s nicer and they’ll still stay portable. It may sound somewhat counterintuitive, and even counter-Oil-philosophy, but it’s kind of a known idea and I’m throwing it in if you’re at a point where you’re thinking about possible paths. Basically, it’s what C did to Assembly, C++ or Nim did to C, etc. etc.

                      As to the rest, I noted to myself to try replacing bash with Oil on my secondary “experimental” personal laptop running NixOS when I find some time for that.

          2. 1

            Re the macOS libc question: this is a limitation due to macOS’s design. On macOS, the only supported way to interface with the kernel is by dynamically linking with the system-provided libc. (Nothing stops you from making raw syscalls, but there’s no guarantee they’ll keep working with OS updates; no idea how common breakage is in practice.) Therefore, even “static” and Nix-built binaries depend on the system libc.

            Incidentally, Linux is the weird one here—Windows and the BSDs also expect all applications to use a system-provided dynamic library.

            Even on Linux, of course, any isolation mechanism short or a full VM relies on the host system to some degree; it’s just a question of whether you draw the line at the kernel-userspace interface or the library-application interface.

            1. 2

              Incidentally, Linux is the weird one here—Windows and the BSDs also expect all applications to use a system-provided dynamic library.

              This isn’t really true. While libc is the way you’re “supposed to” access system calls on Unix-like systems, many operating systems do their best to provide a stable kernel system call interface. Dynamic linking is a relatively new invention compared to Unix as a whole.

              NetBSD is an interesting case - it provides a stable syscall interface, but it’s clearly not supposed to be used directly. Syscalls have interesting, versioned names like __gettimeofday50. Calling just “gettimeofday” without the abstractions and aliases provided by libc and its headers gets you the version with 32-bit time_t, provided only for binary compatibility, whereas __gettimeofday50 is the “real”, 64-bit one.

              I think OpenBSD is the only BSD where making system calls outside of libc is expressly forbidden.

          3. 1

            That sounds like a problem that would exist on any system that isn’t the developer’s one. How are the shells installed? Is it that they don’t build on macOS (and they’re built from source by the test suite)? Or are they built for Linux and downloaded from somewhere? Or are they expected to be installed by the system’s package manager (in which case how do you handle all of the different package management tools on different *NIX systems)?

            My limited experience with Python has been on macOS, where pipenv was able to install the Python version that I needed and the packages that I wanted. I thought (as someone who tries to avoid Python where possible) that this was now the recommended way of developing Python things. Hopefully once the Python to C++ translation is finished the Python dependency for contributors will go away.

            1. 1

              Right now I build specific versions of shells from source, and mirror them on the site: https://www.oilshell.org/blob/spec-bin/

              So there are shell scripts in the repo that simply download and build these shells, which is very reliable on Linux (since shells have very few dependencies). It’s less tested on OS X but in theory some of it may work there. (But again you then have the problem of slightly differing test behavior due to file system layout and libc.)

              I think this problem is pretty universal … e.g. if you work on LLVM or the Rust compiler you’ll have a bunch of dev dependencies too. And how do they get stable versions across OSes? I think they mostly don’t?

              How many of those devs use Windows and OS X ?

              Though perhaps our problems are magnified because shell touches more of the OS – a C compiler perhaps just looks at files in the repo, and the dev tools could be quite portable across OSes. I think LLVM has some problems with the Python version of their tools, e.g. especially during Python 2 to 3 transition.

              There are some commercial services trying to address similar problems – Github code spaces



              I’m not opposed to the cloud OPTIONALLY, but that shouldn’t be the primary way to do it of course.

              Our source language will continue to be Python, but it doesn’t rely on pip or pipenv. Everything is in the repo (except the lint tool which is non-essential)

              I like Python but I dislike its packaging tools! To be honest that is one reason I lean towards shell so much. Because simply downloading Python tarballs with shell scripts was a lot more reliable than pip, at least for awhile! pip has the “who installs the installer” problem and it was not reliable. (This is well known; I see Linux distro developers complain about Python packaging to this day.) The layers of distutils / virtualenv / setuptools / pip is ghastly!

              And a tradeoff of the metaprogramming approach is that there are more tools, so the dev dependency issue is more magnified.

              So Oil isn’t just a plain C project where the tarball looks roughly like the git repo and you can hack away… that is unfortunately a downside. I should actually mention that on this page. Many contributors had a hard time with the difference between the git repo and tarball, as it is significant!

              1. 1

                I think this problem is pretty universal … e.g. if you work on LLVM or the Rust compiler you’ll have a bunch of dev dependencies too. And how do they get stable versions across OSes? I think they mostly don’t?

                LLVM has very few host dependencies. To build, it needs CMake and a C++ toolchain. To build quickly, it wants ninja (brew install cmake ninja or choco install cmake ninja gets me here on macOS or Windows). The tests need Python, but any Python 3 install is fine (Python 2 and 3 were supported in parallel for a while). The system or Homebrew one works on macOS, the one from Chocolatey works fine on Windows.

                How many of those devs use Windows and OS X ?

                A lot use macOS, a few use Windows. At the last LLVM DevMeeting I went to, I’d guess that about half of the laptops were Macs.

                There are some commercial services trying to address similar problems – Github code spaces

                That doesn’t address the problem if you are writing portable code. It locks you in to targeting Linux or Windows containers (but not both) unless you do a lot of extra work on top. You can easily deploy a Linux environment for building, but that doesn’t help a user who wants to use *BSD, macOS, or whatever.

                I generally view lack of a portable build environment as code smell. *BSD, Linux, Solaris, and macOS are the same in far more ways than they are different. If your development process is so tied to Linux (or, worse, to a specific Linux distro) that you can’t easily make any of the others work, then you’re going to have enormous pain when the distro that you’re targeting decides to change one of the things that you depend on. If you have multiple operating systems in CI then I have a lot more confidence that you’re not tied to fragile implementation details.

                Our source language will continue to be Python, but it doesn’t rely on pip or pipenv. Everything is in the repo (except the lint tool which is non-essential)

                I was under the impression that you were writing in a DSL that happened to work with a Python interpreter and would later work with your own tooling.

                1. 1

                  Yeah this is fair, I filed this issue to try FreeBSD / Cirrus CI and OS X / Circle CI which are the platforms I know of…


                  (I actually started this awhile ago because I have a pet project of running shell scripts on as many cloud services as possible. That is, using cloud providers as “dumb data planes”)

                  It will be very easy to get say bin/osh -c 'echo hi' working. It probably already works because the whole thing is plain Python and plain C extensions. It takes 10 seconds to build and is easy to hack on. (Somehow this fact gets lost … https://github.com/oilshell/oil/wiki/Contributing )

                  What is hard is the huge array of tests – https://www.oilshell.org/release/0.10.1/quality.html

                  If FreeBSD / OS X have good container support that would help – i.e. the shell can run outside a container, but the tests should run within one.

                  So that is very doable, but honestly it’s likely to be low priority unless someone really wants to hack on it. The C++ translation is the critical thing now, and I have to do many things to “unblock” that as much as possible. FreeBSD and OS X aren’t on the critical path.

                  The reason I was sort of negative in this wiki page is that there was a pattern where people would get lost in yak shaving on OS X …

                  Like you could spend months fixing up 10,000 lines of shell tests. Or you could push the project forward with user-visible things!

                  And I again emphasize that the tarball is portable. People built and run it on OS X and BSD years ago.

                  Regarding Python, it is a DSL subset, but practically speaking almost all the tools in the repo are shell + Python. The Ninja file is generated by Python, etc. The multiple test frameworks are all Python; mycpp is written in Python, etc.

                  But again I don’t use the standard Python tooling because it’s unstable – that is one of the main motivations for shell :)

                  What I would really like to see from FreeBSD and OS X is some equivalent of https://github.com/containers/bubblewrap or systemd-nspawn etc. That is highly unportable code and complementary to a shell. It would be directly applicable to this problem