1. 40
  1.  

  2. 23

    My colleague and I have resorted to bash on a number of occasions to unfuck tooling and workflows written with “better” systems.

    Like a lead pipe, broken beer bottle, or sharpened toothbrush handle, bash will be there for you in your hour of need.

    1. 13

      I find shellcheck to be invaluable when writing bash scripts. The amount of little gotchas you need to be aware of is staggering and the documentation links it spits out are pretty good as well.

      1. 3

        ShellCheck is a great tool. However I realized that for me, the problem wasnt checking my large Shell scripts. The real problem was having large shell scripts. Shell scripting is such a hard thing to get right, by that I mean POSIX portable. So I ended up rewriting many scripts in PHP or other languages.

        1. 2

          shfmt as well for making scripts look the same.

          EDIT: Fixed incorrect command name.

          1. 1

            Interesting, can you link it? I found https://github.com/mvdan/sh, but that seems to be include a different formatter than the one you mentioned.

            1. 2

              That’s the one!

        2. 9

          Writing good shell scripts takes a lot of discipline and effort if you ask me. On the long run, after some practice it becomes easier though. I disagree with using bash idioms everywhere though. POSIX sh is sufficiently complex to do everything you need, whill keeping things simple enough so you don’t get surprised by bash’s « cool features ».

          An example would be rhe [[ ]] constructs, which can be very misleading:

          chmod 0400 file.txt
          # format %a returns permission in octal mode
          mode=$(stat -c %a file.txt)
          if [[ "$mode" -eq 0400 ]]; then
              echo "File is read-only!"
          fi
          

          I think that all the bash features turn shell scripting into a full language, which it is not. Shell scripting should only be some glue between actual programs, where the actual logic is. Another advice I would give is to try to keep every shell script under 100 lines, so they remain manageable and auditable. More lines usually means that the programs you are wrapping are lacking some features, or are hardly composable.

          1. 1

            Another advice I would give is to try to keep every shell script under 100 lines, so they remain manageable and auditable. More lines usually means that the programs you are wrapping are lacking some features, or are hardly composable.

            Yes and no. My current project has several tens of thousands lines of bash, with some scripts in several thousand line long. And those are just calling external programs and piping them forward, doing simple if statements occasionally.

            1. 1

              An example would be rhe [[ ]] constructs, which can be very misleading:

              chmod 0400 file.txt
              # format %a returns permission in octal mode
              mode=$(stat -c %a file.txt)
              if [[ "$mode" -eq 0400 ]]; then
                  echo "File is read-only!"
              fi
              

              Not sure what you mean by [[ and ]] being misleading in this case? It’s not the fault of bash that you are comparing 0400 (an octal number) with 400 (a decimal number)?

              1. 1

                I mean this:

                bash-5.0.3$ [[ '0400' -eq '400' ]] && echo decimal || echo octal
                octal
                bash-5.0.3$ [ '0400' -eq '400' ] && echo decimal || echo octal
                decimal
                

                Note: I know and understand what’s going on here. But this is definitely misleading as some people tend to use [[ ]] without even thinking about it, or understanding how an if block is supposed to work in shell. For example, I have seen this a huge number of times:

                grep -q 'foobar' file.txt
                if [[ $? -eq 0 ]]; then
                    # do stuff
                fi
                

                This could be (in fully POSIX shell):

                if grep -q 'foobar' file.txt; then
                    # do stuff
                fi
                
                1. 2

                  I would argue that [[ and ]] is doing the least surprising thing here because it is actually evaluating the arguments when in an arithmetic context. It’s the same with $((0400 + 1)) evaluating to 257 and not 401.

                  Interesting, in zsh, $((0400)) evaluates to 400, even though $((0x400)) evaluates to 1024. Now that’s surprising and kinda buggy! But that’s why I don’t write shell scripts in zsh ;)

                  1. 2

                    That’s indeed surprising of zsh because the arithmetic expansion $(( )) is the only place (according to POSIX) where octal numbers are to be interpreted by the shell.

                    Interestingly, mksh does the same as zsh here, and express it explicitely in the manpage:

                    Arithmetic expressions
                    […]
                    Prefixing numbers with a sole digit zero (“0”) does not cause interpretation as octal (except in POSIX mode, as required by the standard), as that’s unsafe to do.

                    As for [[ ]] vs. [ ], this makes sense because [[ ]] is a shell construct, while [ ] is a program (see test(1)). If the shell where expanding numbers with a leading 0 as octal before passing them to programs, we would be fucked.

                    1. 1

                      Thanks for looking that up on mksh. Explains alot. But I wonder why it is unsafe?

                  2. 1

                    Note: I know and understand what’s going on here. But this is definitely misleading as some people tend to use [[ ]] without even thinking about it, or understanding how an if block is supposed to work in shell. For example, I have seen this a huge number of times:

                    grep -q 'foobar' file.txt
                    if [[ $? -eq 0 ]]; then
                        # do stuff
                    fi
                    

                    Yes, this is absolutely horrible and stems from the fact that people do not know that all if is supposed to do is execute its argument and respond to the return value.

                    Some trivia for geeks: if [ 42 -eq 41 ]; then ... is actually just executing /bin/[ with the four arguments "42", "-eq", "41" AND "]". /bin/[ simply ignores ] as syntactic sugar. Example:

                    sh-5.0$ /bin/[ 0 -eq 1 ]; echo $?
                    1
                    
              2. 6

                FTA:

                Has a shebang and that shebang is #!/usr/bin/env bash

                One of my favourite things to do with Nix is use nix-shell as the shebang, so that I can ask for a specific interpreter and the tools I need instead of having my scripts fall over at the first missing command.

                Here is the start of a shell script that transcodes my flacs (for archival) to ogg-vorbis (for my personal music player):

                #!/usr/bin/env nix-shell
                #!nix-shell -i bash -p flac vorbis-tools
                
                1. 4

                  I confess to always putting the following in every run.sh bash script. Despite the obvious simplification available, it’s always in there as-is.

                  # From https://stackoverflow.com/a/246128/3858681
                  DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
                  cd "${DIR}"
                  

                  From this StackOverflow answer

                  1. 5

                    BTW this reminded me that I added readlink to Oil specifically to reduce this boilerplate. But I honestly forgot about it until now!

                    https://github.com/oilshell/oil/issues/96

                    The shorter version with readlink is:

                    DIR=$(dirname $(readlink -f $0))
                    cd "$DIR"
                    

                    But readlink -f isn’t on OS X, which is the reason for all the cd / BASH_SOURCE stuff. And I used to use that myself too.

                    So maybe Oil needs a shortcut for $(dirname $(readlink -f $0)). It already has all the logic for it.

                    Filed issue here: https://github.com/oilshell/oil/issues/587

                    1. 1

                      GNU coreutils also have realpath which I use on my Mac all the time when I’m at the command-line and I need to get an absolute path for something else (like a COPY table FROM into DataGrip for my local postgresql).

                      ~ ➜ /tmp/example/of/cows/script.sh
                      /private/tmp/example/of/cows
                      ~ ➜ cd /tmp/example
                      example ➜ ./of/cows/script.sh
                      /private/tmp/example/of/cows
                      example ➜ cat ./of/cows/script.sh
                      #!/usr/bin/env bash
                      
                      dirname $(realpath $0)
                      

                      It is a bit overweight for this because it canonicalizes and all that which you really don’t need to get to the directory.

                      1. 2

                        If you want to handle pathological cases like commands with spaces in the name or commands named --help you’ll need something more quotes and --, something like DIR="$(dirname "$(realpath -- "$0")")".

                        Example (indentation added for readability): https://paste.stevelosh.com/5517e2b981a88dee24b7ae99dfca0c3974c38d15

                        1. 2

                          If you know you always have coreutils, then you can simplify all the BASH_SOURCE stuff to just

                          DIR=$(dirname $(readlink -f $0))

                          It does the same thing as the longer incantation.

                          I’m not sure if realpath is the same as readlink -f. If it is, then it can be even simpler with $(dirname $(realpath $0)). But I forget.

                          I got used to use the cd trick` because I didn’t have coreutils on OS X.


                          edit: or maybe it’s DIR=$(dirname $(readlink -f ${BASH_SOURCE[0]})). All this stuff is gross and hard to remember :)

                          1. 2

                            Note that readlink isn’t POSIX, and I believe that -f isn’t supported on all platforms with readlink either (I forgot which, I just remember running in to problems or having them reported and avoiding it since).

                            1. 1

                              It’s like you said, I’m on a Mac (which doesn’t have -f) and deploy to Linux on Docker so I just keep it the way it works everywhere out of mental simplicity. It is true that I install coreutils everywhere though like some sort of slob throwing his clothes around the bedroom but the seduction of just copying standard incantation guaranteed to work everywhere (since I use bash everywhere) is just overpowering.

                          2. 2

                            realpath is also on BSD, but it’s not on all Linux systems. I believe it’s missing by default on either Ubuntu, CentOS, or both (you can still install it, of course, and some Linux systems do have it by default).

                      2. 3

                        For the masses of Bash scripts that have built up at work I’ve mostly come to prefer Python. Many scripts do similar things so you want to abstract them so that a change somewhere doesn’t break everywhere. Now you’ve got to got to pass structured data as strings and hope you did it correctly.

                        Parallelism and scripted job control also isn’t easy to get right in Bash (vs. the built-in multiprocessing and subprocess libraries in Python). At least, I’ve done scripted job control in Bash to check log files during service startup, and I haven’t gotten it right yet. For parallelism there’s xargs and gnu parallel. We do use xargs for parallel jobs in places.

                        We use set -euo pipefail in many places. I’ve used Shellcheck in the past but struggled to integrate it with the diversity of scripts at work.

                        All this leads me to generally prefer Python for not-clearly-one-off scripts. It’s safer than Bash, allows you to use “complex” data types, and allows you to parallelize work pretty easily.

                        Some day I plan to reimplement Bash so I can have a genuine understanding of it. But I’d rather not expect that of developers on the team, and Python is generally safe enough.

                        1. 3

                          What’s the point of the installed function testing if it’s a regular file? Couldn’t you just use something like hash "$1" 2>/dev/null or command -v "$1" >/dev/null 2>&1?

                          1. 1

                            Oh, it seems to be to try to exclude aliases and such; hash works for that, or if you also want to exclude builtins, type -P "$1" >/dev/null.

                          2. 2

                            I disagree with this advice. Anything that requires actual data structures should be in python or go or something else. I know bash technically has maps and arrays, but the syntax is awful and if the script is anything more than just run these commands, just, please, write python

                            1. 2

                              Writing good bash becomes surprisingly approachable with these 3 things:

                              1. Start with a sane template (with the set -euo pipefail and other shenanigans).
                              2. Use code snippets (aka live templates) in order to create a loop, a function, or anything really.
                              3. Use ShellCheck.
                                1. 1

                                  One thing I keep preaching no matter what your tool of choice? Code budgets. Each line of code has limited value and potentially unlimited liabilities (because you might have to come back and maintain it). You should have a budget.