1. 2

    What I really want is something like Python (or like Haskell, depending on the circumstances) where running shell commands is a “first-class” language construct, with ergonomic piping of stdout/stdin/files. Oil seems much-improved compared to Bash, but it (intentionally, for good reason) borrows many of the same bad design choices with regard to control flow and cryptic naming schemes.

    The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

    1. 8

      Why not Perl? It’s the obvious choice for a language where shell commands are a “first-class” language construct.

      1. 3

        Totes agreed!

        I find shell logic and data structures impenetrable and error prone, so I write all my “shell scripts” in Perl. It’s easy to pull shell output into Perl data structures using backticks, e.g. print the names of files in $TMPDIR iff they are directories

        chomp(my @out = `ls $ENV{TMPDIR}`); for (@out) { print "$_\n" if -d qq[$ENV{TMPDIR}/$_]; }
        

        For non-perl users, the backticks above mean “run this shell command”, chomp removes the newlines, qq[] means “wrap this in double quotes and handle the annoying escaping for me”, and $ENV{X} is how you access env var X, etc.

        The book Minimal Perl for UNIX and Linux People has a lot of information about how perl can be used to do everything on unices that awk, grep, sed, etc. can do

        also with Perl you get a real debugger (see also perldoc perldebtut or the book Pro Perl Debugging)

        1. 3

          I think it easily gets overlooked these days, since it is no longer hip and therefore less on the frontpage news

          1. 5

            Apropos of “no longer hip”, tcl is a nice scripting language with a clean syntax.

            1. 1

              I used to mess with tcl scripts all the time in the early 2000s when I ran eggdrop irc bots. Good old days…

              1. 4

                Sqlite started as a tcl extension and still has a good tcl interface: https://sqlite.org/tclsqlite.html

                1. -2

                  TIL!

        2. 3

          Depending on your use cases, have you ever tried Fish, or more on the programming side, Raku/Perl? Raku is awesome for shell-style scripting.

          1. 2

            Hm what do you mean by control flow? I’m working on more docs so it probably isn’t entirely clear, but control flow looks like this

            if test --dir $x {
              echo "$x is a dir"
            }
            

            And you can use expressions with ()

            if (x > 0) {
              echo "$x is positive"
            }
            

            For loops are like bash with curly braces

            for x in a b c { echo $x }  # usually written on 3 lines, being compact here
            for x in *.py { echo $x }
            for x in @(seq 3) { echo $x }  # explicit splitting rather than $(seq 3)
            

            I’m also interested in feedback on naming. Oil has long flags like test --dir instead of test -d. I am trying to get rid of all the one letter obscureness (or at least ALLOW you to avoid it if desired).


            It’s accurate to describe Oil as “Python where running shell commands is first class” (and pipelines and more).

            Although I would take care to call it perhaps the “David Beazley” subset of Python :) That is a prominent Python author who has been lamenting the complexity of Python.

            He writes some great code with just numbers, strings, lists, dicts, and functions. Oil is like Python in that respect; it’s not like Python in the sense that it has a meta-object protocol (__iter__ and __enter__ and __zero__ and all that).

            I’m interested in language feedback and if that matches your perception of Oil :) Does the code look cryptic?

            1. 3

              I’ve only had a cursory look at Oil, so forgive me if I have the wrong idea :). I really admire the effort, and I can really relate to what you’ve written about your motivations for creating Oil!

              Oil is also aimed at people who know say Python or JavaScript, but purposely avoid shell

              It seems like Oil will be great for people who spend a lot of time maintaining shell scripts and want to work with something a little more sane. However, the Oil docs impression that Oil will be just as confusing as Bash for someone like me who wants to spend as little time as possible writing / reading / maintaining shell scripts and who only finds occasion to do so once every few months. For someone like me, I wish there were some syntax-level construct in Python or Haskell for running shell commands that lets me take advantage of my existing knowledge of the language, rather than having to learn something entirely new.

              I took another look at the Oil docs, and the control flow actually seems ok, even if minimal. It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

              I think the biggest improvement a new shell-scripting language can offer is an alternative syntax for specifying command-line flags. Now that I think of it, this is actually the main reason I wrap all my CLI calls in Python scripts. For instance, here are some commands I had to run at work last week:

              docker run -itd --name container_name --rm -e CUDA_VISIBLE_DEVICES=0 -e API_KEY="..." -v /home/ben:/ben/ -v /data/v2/:/data image_name
              
              docker exec -d container_name sh -c 'mkdir -p /ben/train/train_2021-05-16_144614 && python /ben/project_name/train.py \
                              --batchSize 16 --loadSize 160 \
                              --alpha 1.0 \
                              --annotations /data/project_name/sourceA/v3/really_long_name.txt \
                              --annotations /data/project_name/sourceB/v1/really_long_name.txt \
                              --annotations /data/project_name/sourceC/v4/really_long_name.txt \
                              --data-path /data/project_name/images \
                              --output-path /ben/train/train_2021-05-16_144614/checkpoints \
                              --gpu_ids 2 \
                              2>&1 > /ben/train/train_2021-05-16_144614/log_2021-05-16_144614_outer.txt'
              

              I actually had to run the exec command four different times with different arguments. I wrote a Python script to generate the Python command, wrap it in the docker command, and pass in all the necessary parameters with string interpolation. But to me this seems like a pretty silly to glue together different programs, when we already have familiar syntax for function calls! I would love to write instead:

              var env_variables = [
                CUDA_VISIBLE_DEVICES=0,
                API_KEY="..."
              ];
              
              var volumes = [
                /home/ben/:/ben/,
                /data/commondata/v2, /data
              ];
              
              docker.run(
                flags="interactive, tty, remove",
                env=env_variables,
                name="container_name",
                image="image_name",
                volume=volumes
              );
              
              ...
              
              image_annotations=[ /data/project_name/sourceA/v3/really_long_name.txt, ... ]
              
              python_command=python(train.py,
                batchSize=16, loadSize=160, alpha=1.0,
                annotations=image_annotations,
                data-path=/data/project_name/images,
                output_path=...
                gpu_ids=2,
              )
              
              docker.exec(
                detached=True,
                container=container_name,
                command=python_command
              )
              

              This should work, automatically, for ANY command line interface, not just docker. It should work without writing any custom wrapper code. Without actually running the command, I want some kind of TypeScript-style static type-checking to know if the flag/argument I provided does not match what the CLI expects.

              1. 1

                Thanks this is useful feedback!

                Yes I understand that Oil feels like it’s for the “shell expert” now. Because I do frame many things in terms of an upgrade path from bash, and that requires knowing bash (lots of people don’t!).

                But it’s absolutely the goal for a Python/JS programmer to be able to pick up Oil with minimal shell knowledge. You have to understand argv, env, stdin/stdout/stderr, and exit code, and that’s it hopefully. Maybe file descriptors for the advanced.

                I have an upcoming “Tour of Oil” doc that should help with this. It will document Oil without the legacy stuff.

                It would be cool to see some more tools to map / flatmap / fold commands, especially when I have a bunch of parameters that are used to generate a sequence of commands.

                Yes, great feedback, there is a pattern for this but it’s probably not apparent to newcomers. In shell map / flatmap are generally done by filtering streams of lines, and I think that pattern will help.

                https://github.com/oilshell/oil/issues/943

                There is also splicing of arrays, which are natural for command line args

                var common = %( --detached=1 --interactive )
                var args1 = %( --annotations /data/1 )
                var args2 = %( --annotations /data/2 )
                
                # Run 2 variants of the command
                docker run @common @args1
                docker run @common @args2
                
                # Run 4 variants
                for i in @(seq 4) {
                  docker run @common --index=$i
                }
                

                Does that make sense as applied to your problem? You can also apply to env vars with “env”:

                var env_args = %( PYTHONPATH=. FOO=bar )
                env @env_args python -c 'print('hi')
                

                There is no static checking, but Oil’s philosophy is to give errors earlier in general. It will be possible to write your own argv validators if you like.

                A useful pattern will be to define “subsets” of Docker as “procs”. As in, you can just accept the flags you use and validate them:

                proc mydocker {
                   argparse @ARGV %OPT {   # not implemented yet
                      -v --verbose "Verbose flag"
                   }
                   docker @myarray
                }
                mydocker --oops  # syntax error
                

                Thanks for the feedback, let me know if that makes sense, and if you have other questions. And feel free to follow the issue on Github!

            2. 1

              Golang exec’s module is much better for piping in the classic shell way. You might consider that.

              1. 1

                The best solution I’ve found so far is to call subprocess.run from a Python script and use string interpolation to pass arguments to the commands I want to run. But this is far from ideal.

                I do a similar thing. What sort of improvements can you envision?

              1. 5

                Awesome! I going to use this to write the RSS reader that I always visualized. I’m using Miniflux right now, but I just wish it were more information dense.

                1. 6

                  Have you tried the reminiflux frontend? It has a more traditional three-pane layout that is quite dense.

                  1. 1

                    Woah, I didn’t know this was a thing! Neato.

                1. 2

                  I once wondered why some Google API returned structures like {'lo': 123, 'hi': 0} instead of just the number 123, and realized that the numbers probably can grow beyond 2**53.

                  1. 12

                    One downside of using the normal Unix utilities for csv files is that handling quoted fields is complicated. If you have a comma inside a field, the field needs to be quoted.

                    I’ve used xsv table, and xsv includes other useful commands for csv files that you can pipe together: https://github.com/BurntSushi/xsv

                    1. 2

                      Yeah, quoting is one reason I started using TSV over CSV.

                      There’s an RFC that actually defines TSV not to contain tabs! So you can’t express them at all. But at least it’s consistent, unlike CSV. I have seen some really horrible things people call “CSV”, e.g. emitted by hardware!


                      So I have an extension planned which I’m now calling QTSV: https://github.com/oilshell/oil/wiki/TSV2-Proposal

                      It’s very simple in that you’re allowed to put QSN in cells: http://www.oilshell.org/release/latest/doc/qsn.html

                      So a QTSV file is syntactically a TSV file, and it will work fine with column, and be able to express tabs in fields.

                      1. 3

                        I wonder why tooling doesn’t emit the record separator and friends, which are specified by ASCII…

                        1. 2

                          I can think of a couple reasons:

                          • text editors don’t support them well. If tools don’t support them, then they might as well not exist. Imagine if text editors didn’t support the \n character to mean the end of line – would people still use it?
                          • ditto for terminals. It’s nice to be able to view a TSV or CSV file in a termial.
                          • grep doesn’t work anymore, since it’s inherently line-based.
                          • You still have an escaping problem. Just like TSV fields can’t contain tabs, the RS delimited files can’t have fields with RS. So it just shifts the problem around.

                          (QTSV doesn’t have these problems. Being able to just grep is a feature, but you could also make your own qtsv-grep, etc.)

                      2. 2

                        One downside of using the normal Unix utilities for csv files is that handling quoted fields is complicated. If you have a comma inside a field, the field needs to be quoted.

                        Yes, this is a classic problem. Fortunately CSV is one of formats supported by Relational pipes, so you can do e.g.

                        cat data.csv | relpipe-in-csv | relpipe-out-tabular | less -RSi
                        

                        and see the well-parsed CSV content. Or do some filtering or transformation using relpipe-tr-sql, relpipe-tr-awk, relpipe-tr-cut, relpipe-tr-grep, relpipe-tr-sed, relpipe-tr-scheme etc. You can also convert data from different formats to valid CSV using relpipe-out-csv.

                        1. 1

                          FWIW I’ve had relational pipes on my wiki for awhile:

                          https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

                          Although I don’t think the use of binary formats plays well with shell-like tools. Also it seems there is not a spec, e.g. how do you escape the record separator, etc.

                        2. 1

                          Any idea why we do not use escaping for commas that are not part of the field spec?

                          sentence, here is an escaped\, comma, and more

                          1. 1

                            My guess is that someone had the idea to separate numeric and string fields by enclosing strings in double quotes. This is supported by a SuperCalc manual from 1983 (linked from the Wikipedia page for CSV).

                            1. 1

                              Quotes are also used to embed newlines into a field, so you’d need some mechanism for escaping those as well.

                              1. 1

                                Indeed. But I wonder if the same mechanism can be used

                                 sentence, here is an escaped\
                                 newline, and more
                                
                                1. 1

                                  Yeah, I think it could. There might be some trickiness involved in handling CRLF though.

                          1. 6

                            A related idea: molly-guard asks you to input the hostname when you run shutdown over an ssh session.

                            1. 3

                              This is where the bad-keyboard MBPs are nice: they come with a retroactive four-year keyboard replacement guarantee, and because everything is glued together you don’t just get a new keyboard, you get a fresh battery and I’m sure they blow out the dust too. I’ve had the keyboard replaced twice now on my 2017 laptop, and there’s still time for one more refresh once the keys stop working.

                              1. 1

                                I just heard that the local Macbook repair company has been given stricter rules from Apple: apparently it used to be enough to say “the keyboard doesn’t work right”, but now you have to list the keys that aren’t working, and they’ll first try to clean them with pressurized air. There also can’t be any signs of intentional damage. I guess they suspect people have been abusing the replacement program to get new batteries.

                                1. 1

                                  Yep, I had this repair done last week and the service report includes:

                                  Quote and Assessment Summary
                                  Verified issue reported: YES
                                  Symptom/s: Sticky > key
                                  Ran Apple diagnostic MRI: Passed
                                  Removed, Cleaned, and Replaced effected keycap, issue persists
                                  Visual inspection shows no evidence of liquid contact or corrosion
                                  Isolated to faulty keyboard
                                  Resolution: Replace Top Case
                                  Warranty Status: Service eligible under Apple Limited Warranty, AppleCare, and/or Australian Consumer Law.
                                  Quote to Repair device: $0.00 Customer Approved: Yes
                                  
                                2. 1

                                  I didn’t know that, cheers!

                                1. 18

                                  TIL. I would never expect right-click Open to do anything different to double-clicking the icon.

                                  1. 27

                                    Sometimes MacOS feels like the point-and-click adventure games of the 1990s. The way to see all the resolutions supported by an external display is to Option-click on a radio button in System Preferences.

                                    1. 6

                                      Oof, I just tried pressing Option in the Displays prefpane and a “Detect Displays” button magically appears and disappears. What else.

                                      1. 3

                                        Hold Option when clicking on “Scaled” in the resolution chooser. I think it only works on external displays.

                                    2. 6

                                      I feel like MacOS has a weird history of GUI actions being “special” in these weird ways. Reminds me a bit of how for a while Windows wasn’t great at offering good non-GUI alternatives for actions, with the added obfuscation that Apple doesn’t really need to do much for enterprise vendors relative to Microsoft

                                    1. 12

                                      I haven’t used it in production, but PostgREST uses the database-level access control features:

                                      PostgREST handles authentication (via JSON Web Tokens) and delegates authorization to the role information defined in the database. This ensures there is a single declarative source of truth for security. When dealing with the database the server assumes the identity of the currently authenticated user, and for the duration of the connection cannot do anything the user themselves couldn’t.

                                      https://github.com/PostgREST/postgrest

                                      1. 1

                                        I use it in a side project, and it’s great so far!

                                      1. 3

                                        Attending some of the JuliaCon workshops since I’ve been meaning to learn Julia and this seems like a good opportunity.