1. 23
  1.  

  2. 11
    1. One small and typical problem area is that the Bourne shell doesn’t have named function arguments
    1. “outsourced language elements”
    • test and expr functionality is all in Oil
    1. The third problem is that the Bourne shell lacks important language features that normally act to make coding errors less likely and contain and manage code complexity
    1. Lacking data structures does a number of bad things
    • Oil fixes both bash data structures (arrays and associative arrays) and has recursive Python/JS-like data structures, with better syntax

    If you already have 3400 lines of shell like dkms and don’t want to rewrite it [1], Oil is a good upgrade path. (It still needs to be faster though, working on it)

    https://github.com/dell/dkms/blob/master/dkms

    If you don’t agree after trying it, send me some feedback: https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback

    [1] trivia: I spent more than 2 years rewriting 10K lines of shell as 30K lines of Python early in my career :)

    1. 9

      This is an obvious promotion of the tool this person works on.

      1. 31

        Is that an issue?

        @andyc correctly points out what he thinks are weaknesses and how an open source software he’s been decimating all his free time is trying to solve.

        I find both the list of issues interesting and the attempt to fix them useful. Even if it feels like self-plug, I always learn from his posts (thanks for that @andyc btw).

        Some other people on Lobsters are always posting about how their favorite software is solving some issues. Since they aren’t the creator of the software, less people notice it and nobody seems to complain about that.

        1. 14

          It is for me. I constantly see posts from the dev (comments too) to the point where it feels like spam.

          On top of that, I’m constantly seeing very stuck up views about “fixing” issues with other languages and shells or making this “better”. Those are very subjective things, “better” and “fixing” are nowhere near the right words.

          1. 16

            The original post claims that these are problems, and I agree. If you disagree, then writing a top-level comment about it would be a useful contribution.

            Writing a disagreement to Oil is also useful – in fact that’s why I post the message, so people can disagree and the shell can become better. I explicitly ask for feedback.

            On the other hand, vague disagreements about “spam” aren’t useful.

            1. 9

              Allegations of spam can be useful if you reflect on them. These allegations are constructive criticism, not of your project itself but of the optics you’re giving it, which are also important for adoption.

              Here’s my anecdote. I looked at Oil shell a long time ago, decided that it wasn’t for me on so many levels that I didn’t even see a point in arguing with you. After that, I constantly saw your comments about your shell hijacking threads about shell scripting with self-promotions. After some point, I stopped reading comments authored by you altogether, as I expected them by default to contain some rehash of how you do error handling, or data structures, or whatnot, which I already knew about, having tried your shell and having read some earlier rehash. So, in a way, I react to your comments in the same way I react to spam, by skipping them, mildly annoyed.

              If, on the other hand, I expected your posts to provide some information that is not yet known to any regular reader of threads about shell scripts, I would be reading them, as you do provide interesting insights from time to time. So, in a way, if you posted more rarely and succinctly, I would know more about what happens with Oil, and maybe be more inclined to look at it again. As is, I have zero interest in hearing about it.

              I don’t claim at all that my anecdote is representative of all the potential readers, and I don’t claim that you need to change your behavior. I’m writing this because I believe that you’re being very unfair when dismissing feedback from other users of a site that you frequent: the impression that you leave does matter to your project. And I am sure that giving you that feedback took resolve and courage, as your behavior is obviously well-intended, and thus is difficult to criticize! Just looking at number of upvotes that @Phate6660’s post has may tell you that this criticism is something to consider, not disregard as vague.

              1. 6

                Thanks for the reasoned response. Yes I do appreciate that a number of people have found some comments offputting or arrogant.

                I suppose my best defense is that I have been a bit “short” in comments because I have had these conversations about “shell is not suitable for X” many many times, and I am building stuff to make it suitable for X. But yes I realize being “short” does not help the situation, so I will try to change that.

                However I also struggle to understand what else I should do differently, since I don’t understand the feedback. I do tend to dismiss “vague” criticism and I appreciate specific criticism more. For example:

                I looked at Oil shell a long time ago, decided that it wasn’t for me on so many levels that I didn’t even see a point in arguing with you. After that

                That’s fair but I don’t understand what the problem is. What are those levels? Why? Sometimes people don’t like things, sure, but it would help to know the substance of the disagreement.

                Reading between the lines, my guess is: Oil is an upgrade from bash and thus full of details that seem irrelevant, and I don’t want to learn them. That is fair, though my response is:

                1. It would be nice for people to state this explicitly, or state what the real problem is
                2. This is mostly a problem of the way it’s documented now – as an upgrade from bash. It’s also intended to be a “clean slate” language for Python or JavaScript programmers, and I think it’s succeeding. As mentioned there is an upcoming Tour of Oil that will address this.

                So, in a way, I react to your comments in the same way I react to spam, by skipping them, mildly annoyed.

                To me that is fair, not everyone is interested in all content, but the site already has mechanisms for filtering that. I’m not interested in many stories about Rust, and on occasion I simply “hide” them. That’s working as intended IMO.

                If people don’t like my comments and stories then they won’t upvote them. And to be fair I have taken feedback from the people upvoting the complaints.

                I’d be interested in complaints about specific comments rather than just complaints about frequency. IMO Oil is absolutely on topic for this thread. I did not reply with generic advertisements for Oil. I read the article and replied to exactly the OP’s complaints about shell, and responded with Oil’s solutions.

                Likewise in other topics I often point people to https://github.com/oilshell/oil/wiki/Alternative-Shells and the like – I don’t mention Oil only. I spent a lot of time making the wiki page so people know about alternatives. After surveying them, I think Oil is the more realistic upgrade path from bash – which this article is about!!! – but other people might have different criteria and choose something else.

                So bottom line is that I will try to be less short, and respond to specific complaints. But I’m still pretty opinionated about things – that is the nature of anyone building something – and I invite disagreements with those opinions.

                However I don’t find the “spam” comments actionable. To me that just means I am talking about something that a particular person doesn’t care about, and that’s fine – it can be downvoted or whatever. But I think such comments are valid use of the site.

                If someone responds in C++ threads about Rust that can be annoying, but often it’s a valid use of the site. I draw a distinction between vague boosterism, and informed comparisons. The latter are welcome, even if repetitive. For example, I even responded with Rust in a C++ thread here, because it’s absolutely relevant:

                https://lobste.rs/s/3ls4jj/it_compiles_does_not_always_mean_it_works#c_kqamta

                So I predict you will hear more about Oil in more threads about shell, because it is relevant. I will take specific feedback and try to be more accomodating of the people who have a disagreement with Oil. So far I don’t understand what that is besides “it seems too complex”, and as mentioned I’m working on fixing that.

                1. 0

                  Oil is an upgrade from bash and thus full of details that seem irrelevant, and I don’t want to learn them.

                  Completely missed the mark here. I already know bash, its details don’t bother me.

                  It’s also intended to be a “clean slate” language for Python or JavaScript programmers

                  I am neither. My main language is Haskell, with a spec of Rust, which are, in a way, polar opposites of Python and JavaScript. The main qualities that I seek in languages are simplicity, orthogonality, and hand-holding; Python, bash, and JavaScript—and, likewise, your shell—are complex, cluttered and anything-goes. I can elaborate if you want me to, but you can probably guess what input to expect from people from my crowd.

                  However I don’t find the “spam” comments actionable.

                  A possible way to make your comments look less spammy is to make Oil less prominent in them and instead focus on the interesting parts of the problem that it solves. You, having the domain expertise, probably have more than enough interesting quirks to share.

                  So, instead of

                  Oil fixes shell’s error handling

                  it could be “Fixing the error handling in shells is difficult, which is ensured by the legacy of the current shells. Usually, when a program finishes with a non-zero error code, it means that it failed—until it doesn’t mean that. Program writers became accustomed to error codes not having an immediate effect and use the error codes for all sorts of things. A prominent example is unzip, which mysteriously returns 1 when it encounters any warnings even if they are completely benign, but also when they are not benign at all. In essence, the authors of unzip rely on you reading the stderr anyway. So, enabling the equivalent of set -e by default in a modern shell could have bewildering consequences.”

                  or “Even the error handling that bash has is broken, as it is possible for everything to finish successfully even with set -e. Always remember to at least enable pipefail, folks, and don’t do export and = on the same line! shellcheck warns about the latter, but you can use Oil for a more thorough check of whether your scripts can potentially lose errors!”

                  This would give specific insights and specific reasons to try out Oil, gives credibility to your opinions and, by proxy, to your shell.

                  If someone responds in C++ threads about Rust that can be annoying

                  It can be, and sometimes it feels spammy as well to me. It can be also formulated to be useful for every participant if they highlight the ways for C++ programmers to structure programs in a more robust way. Instead of, “this is fixed in Rust”, it should be “a good way to avoid such errors is by the following pattern; by the way, Rust enforces that you employ it”. This way, instead for attempting to sway readers to go to a different ecosystem, everyone wins, gaining knowledge and insights about the problem area.

            2. 6

              The dev, having worked in this domain for a while with expertise in shell which I would reasonably presume to be far above the average hacker’s knowledge should, for what it’s worth, be able to describe shell design features or flaws that are better or needing to be fixed respectively - following through the links, andyc always explains with examples why such feature was added, changed and such and in what ways to keep Oil shell Bash-compatible. To me that seems like a valuable contribution to any discussion on Bash.

            3. 6

              Thanks, glad you you find the posts and comments useful!

              The main goal is to get more feedback in Oil. Some people might disagree about what the right behavior is, and that’s the point :) I want to hear the disagreement.

              So far I’ve heard mostly agreement, e.g. here today, but non-vague disagreement is valuable!

          2. 3

            You always mention the upgrade path thing, but the documentation doesn’t really make obvious how this works. I understand that osh is (almost completely) bash compatible. But where does one go after changing the shebang line?

            Do you have a doc for that?

            1. 4

              Good question, I will write something up and post it here!

              It will look something like this:

              1. Change the shebang line as you say. This may uncover some problems (e.g. ble.sh found arithmetic expression bugs when running under OSH)
              • You can use a bunch of features like var, const, setvar, proc now
              1. Add shopt --set oil:basic to the top of your program. This may uncover more problems, and it will also unlock even more new features. Probably the biggest change is quoting / simple word eval.
              1. 3

                I started a doc here: https://github.com/oilshell/oil/wiki/Gradually-Upgrading-Shell-to-Oil

                Let me know what you think! There are three steps, and they are intended to be smooth, and give you immediate benefit in exchange for the work involved.

                1. 1

                  This was great! I’m vastly more likely to try osh now

            2. 3

              The first problem is that shell scripts are more or less constrained to be all in a single file. 4,000 lines in a single file is hard to keep track of in any language; people do much better when we can chunk up complexity into smaller units.

              The end result has to be a single file (usually), but there’s no reason it can’t be developed in chunks that are cat’ed together at build time. With proper naming conventions and separation into functions you’d run into no more issues than a comparable C program.

              1. 4

                For what ever it’s worth. It’s not really true that it has to be one file.

                I’ve worked among many many thousand line bash “code”-bases (don’t ask) and we composed common libraries of helper functions which were sourced in where needed.

                1. 2

                  For what ever it’s worth. It’s not really true that it has to be one file.

                  Yeah, the possibility of sourcing functions in when needed was why I added the “usually”. I’m not familiar with any common shell scripts that keep helper scripts/libraries to be sourced in /usr/lib or /usr/libexec, but I’d be happy to be shown any examples.

                  1. 2

                    In FreeBSD, rc scripts source a script called rc.subr, which is a library of useful sh functions provided by the system.

                    1. 2

                      Oh, that’s a good one. I use OpenBSD regularly, which does exactly the same thing, so it’s a bit embarrassing that I didn’t think of it.

              2. 2

                One of my first jobs involved maintaining (ultimately replacing) numerous amount of shell scripting and I can say most everything rings true:

                1. You generally do “outsource” a lot of platform specific stuff to some other utility.
                2. Modularizing shell scripting is difficult outside of inventing your own concatenation build tool so scripts usually do grow in size.
                3. Testing? I really don’t most times, and neither did anyone I worked with, outside of copy/paste snippets into a terminal and see what happens.
                4. Portability is a lie; I think the dkms example demonstrates that clearly with the 10s of different ways to even invoke kernel-install.
                5. Everyone, and I mean everyone, has their own particular shell scripting style; even in the dkms example linked in the article, I see usages of sed, grep, etc that I probably would have done in an inline awk instead.

                What really disturbs me about 4 is I don’t have a good answer when people ask how to write shell scripts. Maybe that’s why everyone comes up with their own quirky way. Usually I just say, read man bash or make a project with it. It if wasn’t so embarrassing, maybe I’d point them to my ridiculous IRC bot written in bash.

                1. 2

                  What I really want is something like Python (or like Haskell, depending on the circumstances) where running shell commands is a “first-class” language construct, with ergonomic piping of stdout/stdin/files. Oil seems much-improved compared to Bash, but it (intentionally, for good reason) borrows many of the same bad design choices with regard to control flow and cryptic naming schemes.

                  The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

                  1. 8

                    Why not Perl? It’s the obvious choice for a language where shell commands are a “first-class” language construct.

                    1. 3

                      Totes agreed!

                      I find shell logic and data structures impenetrable and error prone, so I write all my “shell scripts” in Perl. It’s easy to pull shell output into Perl data structures using backticks, e.g. print the names of files in $TMPDIR iff they are directories

                      chomp(my @out = `ls $ENV{TMPDIR}`); for (@out) { print "$_\n" if -d qq[$ENV{TMPDIR}/$_]; }
                      

                      For non-perl users, the backticks above mean “run this shell command”, chomp removes the newlines, qq[] means “wrap this in double quotes and handle the annoying escaping for me”, and $ENV{X} is how you access env var X, etc.

                      The book Minimal Perl for UNIX and Linux People has a lot of information about how perl can be used to do everything on unices that awk, grep, sed, etc. can do

                      also with Perl you get a real debugger (see also perldoc perldebtut or the book Pro Perl Debugging)

                      1. 3

                        I think it easily gets overlooked these days, since it is no longer hip and therefore less on the frontpage news

                        1. 5

                          Apropos of “no longer hip”, tcl is a nice scripting language with a clean syntax.

                          1. 1

                            I used to mess with tcl scripts all the time in the early 2000s when I ran eggdrop irc bots. Good old days…

                            1. 4

                              Sqlite started as a tcl extension and still has a good tcl interface: https://sqlite.org/tclsqlite.html

                              1. -2

                                TIL!

                      2. 3

                        Depending on your use cases, have you ever tried Fish, or more on the programming side, Raku/Perl? Raku is awesome for shell-style scripting.

                        1. 2

                          Hm what do you mean by control flow? I’m working on more docs so it probably isn’t entirely clear, but control flow looks like this

                          if test --dir $x {
                            echo "$x is a dir"
                          }
                          

                          And you can use expressions with ()

                          if (x > 0) {
                            echo "$x is positive"
                          }
                          

                          For loops are like bash with curly braces

                          for x in a b c { echo $x }  # usually written on 3 lines, being compact here
                          for x in *.py { echo $x }
                          for x in @(seq 3) { echo $x }  # explicit splitting rather than $(seq 3)
                          

                          I’m also interested in feedback on naming. Oil has long flags like test --dir instead of test -d. I am trying to get rid of all the one letter obscureness (or at least ALLOW you to avoid it if desired).


                          It’s accurate to describe Oil as “Python where running shell commands is first class” (and pipelines and more).

                          Although I would take care to call it perhaps the “David Beazley” subset of Python :) That is a prominent Python author who has been lamenting the complexity of Python.

                          He writes some great code with just numbers, strings, lists, dicts, and functions. Oil is like Python in that respect; it’s not like Python in the sense that it has a meta-object protocol (__iter__ and __enter__ and __zero__ and all that).

                          I’m interested in language feedback and if that matches your perception of Oil :) Does the code look cryptic?

                          1. 3

                            I’ve only had a cursory look at Oil, so forgive me if I have the wrong idea :). I really admire the effort, and I can really relate to what you’ve written about your motivations for creating Oil!

                            Oil is also aimed at people who know say Python or JavaScript, but purposely avoid shell

                            It seems like Oil will be great for people who spend a lot of time maintaining shell scripts and want to work with something a little more sane. However, the Oil docs impression that Oil will be just as confusing as Bash for someone like me who wants to spend as little time as possible writing / reading / maintaining shell scripts and who only finds occasion to do so once every few months. For someone like me, I wish there were some syntax-level construct in Python or Haskell for running shell commands that lets me take advantage of my existing knowledge of the language, rather than having to learn something entirely new.

                            I took another look at the Oil docs, and the control flow actually seems ok, even if minimal. It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

                            I think the biggest improvement a new shell-scripting language can offer is an alternative syntax for specifying command-line flags. Now that I think of it, this is actually the main reason I wrap all my CLI calls in Python scripts. For instance, here are some commands I had to run at work last week:

                            docker run -itd --name container_name --rm -e CUDA_VISIBLE_DEVICES=0 -e API_KEY="..." -v /home/ben:/ben/ -v /data/v2/:/data image_name
                            
                            docker exec -d container_name sh -c 'mkdir -p /ben/train/train_2021-05-16_144614 && python /ben/project_name/train.py \
                                            --batchSize 16 --loadSize 160 \
                                            --alpha 1.0 \
                                            --annotations /data/project_name/sourceA/v3/really_long_name.txt \
                                            --annotations /data/project_name/sourceB/v1/really_long_name.txt \
                                            --annotations /data/project_name/sourceC/v4/really_long_name.txt \
                                            --data-path /data/project_name/images \
                                            --output-path /ben/train/train_2021-05-16_144614/checkpoints \
                                            --gpu_ids 2 \
                                            2>&1 > /ben/train/train_2021-05-16_144614/log_2021-05-16_144614_outer.txt'
                            

                            I actually had to run the exec command four different times with different arguments. I wrote a Python script to generate the Python command, wrap it in the docker command, and pass in all the necessary parameters with string interpolation. But to me this seems like a pretty silly to glue together different programs, when we already have familiar syntax for function calls! I would love to write instead:

                            var env_variables = [
                              CUDA_VISIBLE_DEVICES=0,
                              API_KEY="..."
                            ];
                            
                            var volumes = [
                              /home/ben/:/ben/,
                              /data/commondata/v2, /data
                            ];
                            
                            docker.run(
                              flags="interactive, tty, remove",
                              env=env_variables,
                              name="container_name",
                              image="image_name",
                              volume=volumes
                            );
                            
                            ...
                            
                            image_annotations=[ /data/project_name/sourceA/v3/really_long_name.txt, ... ]
                            
                            python_command=python(train.py,
                              batchSize=16, loadSize=160, alpha=1.0,
                              annotations=image_annotations,
                              data-path=/data/project_name/images,
                              output_path=...
                              gpu_ids=2,
                            )
                            
                            docker.exec(
                              detached=True,
                              container=container_name,
                              command=python_command
                            )
                            

                            This should work, automatically, for ANY command line interface, not just docker. It should work without writing any custom wrapper code. Without actually running the command, I want some kind of TypeScript-style static type-checking to know if the flag/argument I provided does not match what the CLI expects.

                            1. 1

                              Thanks this is useful feedback!

                              Yes I understand that Oil feels like it’s for the “shell expert” now. Because I do frame many things in terms of an upgrade path from bash, and that requires knowing bash (lots of people don’t!).

                              But it’s absolutely the goal for a Python/JS programmer to be able to pick up Oil with minimal shell knowledge. You have to understand argv, env, stdin/stdout/stderr, and exit code, and that’s it hopefully. Maybe file descriptors for the advanced.

                              I have an upcoming “Tour of Oil” doc that should help with this. It will document Oil without the legacy stuff.

                              It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

                              Yes, great feedback, there is a pattern for this but it’s probably not apparent to newcomers. In shell map / flatmap are generally done by filtering streams of lines, and I think that pattern will help.

                              https://github.com/oilshell/oil/issues/943

                              There is also splicing of arrays, which are natural for command line args

                              var common = %( --detached=1 --interactive )
                              var args1 = %( --annotations /data/1 )
                              var args2 = %( --annotations /data/2 )
                              
                              # Run 2 variants of the command
                              docker run @common @args1
                              docker run @common @args2
                              
                              # Run 4 variants
                              for i in @(seq 4) {
                                docker run @common --index=$i
                              }
                              

                              Does that make sense as applied to your problem? You can also apply to env vars with “env”:

                              var env_args = %( PYTHONPATH=. FOO=bar )
                              env @env_args python -c 'print('hi')
                              

                              There is no static checking, but Oil’s philosophy is to give errors earlier in general. It will be possible to write your own argv validators if you like.

                              A useful pattern will be to define “subsets” of Docker as “procs”. As in, you can just accept the flags you use and validate them:

                              proc mydocker {
                                 argparse @ARGV %OPT {   # not implemented yet
                                    -v --verbose "Verbose flag"
                                 }
                                 docker @myarray
                              }
                              mydocker --oops  # syntax error
                              

                              Thanks for the feedback, let me know if that makes sense, and if you have other questions. And feel free to follow the issue on Github!

                          2. 1

                            Golang exec’s module is much better for piping in the classic shell way. You might consider that.

                            1. 1

                              The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

                              I do a similar thing. What sort of improvements can you envision?