1. 24

I’ve just released a new version of shfmt, a shell code formatter, along with its Go module including a Bash parser and interpreter.

shfmt now includes a manpage, a filename flag for stdin, support for Bats, position overflow protection, better ksh support, and many bug fixes. Any input welcome!

  1.  

  2. 2

    Thanks for your effort, even if I don’t care for shell formatters.

    Recently I thought about using your module in order to parse the shell’s interactive command line contents and adjust relative paths to match a different origin CWD so that my directory navigator’s shell integration works more seamlessly. It seemed laborious but generally doable (even if it would work only for trivial cases, yet that’s what most command lines are).

    1. 1

      Happy to give tips if you’re interested. The only immediate problem I see is that calling Go from C++ might be tricky, because the parser produces the syntax tree structure. I’m not sure what I could do to make the syntax package more accessible from other languages. There’s shfmt -tojson too, which adds some overhead but should work from anywhere.

      1. 3

        It doesn’t need to be the same program, or even shouldn’t be–the integration’s chdir needn’t succeed, so I have to launch another instance of something anyway–who cares what that’s written in. But you’ve made me realise I’ve started showing the external command line internally–again nothing popen() can’t solve.

        I’ll remember you if I ever get back to it.

    2. 2

      Nice work!

      How far away is the interpreter from becoming a usable replacement for Bash? I wanted to introduce a native bash replacement in direnv in order to remove the Bash dependency. The last time I tried, it didn’t handle all of the features that are being used in the stdlib.

      1. 2

        Looking over stdlib, I don’t see anything that shouldn’t run under the latest release of Oil. I’d be interested in a bug report if it doesn’t run.

        I just ran osh -n stdlib.sh and it spits out a big AST, so that’s a good sign.

        1. 2

          Oil is also interesting and on my radar. For the direnv use-case, gosh has an inherent advantage since it’s written in the same language as direnv.

          My hope for Oil is that we will get something that is faster and stricter than Bash. The new language seems interesting but ultimately what I want is more like a stricter and faster version of Bash. And then rebuild all of nixpkgs with it. Due to the nature of Nix and how the derivations are being composed using chunks of strings, Bash is quite ideal as a builder companion. But the feedback that Bash gives is sometimes lacking.

          1. 2

            what I want is more like a stricter and faster version of Bash

            Ah OK, I didn’t quite get the Go connection, but Oil is definitely a stricter version of bash now, and it’s on track to be faster (maybe 3 to 9 months from now).

            In the last couple releases I really nailed the strictness down. Some examples:

            http://www.oilshell.org/blog/2020/10/big-changes.html#tightened-up-string-literals

            As of early this year, e.g. http://www.oilshell.org/blog/2020/06/release-0.8.pre6.html, I consider the OSH part mostly done. That is, the semantics of the language.

            I would say OSH is the most bash compatible shell by a mile… If that’s what you want, I would think it’s pretty much your only option other than bash itself. (I haven’t tested shfmt recently, but the Go runtime places some fundamental limitations as mentioned)

            There are a few remaining issues, which are mostly unimplemented stuff that existing scripts haven’t tickled. As for divergences, there’s an IFS compatibilty bug I know about, but nobody has hit it in practice.

            https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3Acompatibility

            https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3Adivergence

        2. 1

          Oh neat, I never noticed direnv was written in Go!

          For most Bash code out there, it should be pretty complete modulo some minor bugs you might uncover that would be easy to fix. Some of the funkier Bash features aren’t fully implemented, but if people do use them in the wild I’m happy to take a look or review PRs.

          The only other thing to keep in mind is that this is pure Go, so there’s no real process forking for subshells. This is fine for the vast majority of vanilla Bash code, but if a bash script relies on PIDs or procfs, then the behavior will change with the Go interpreter. The upside, beyond dropping Bash as a dependency, is that you have a tighter control over the interpreter.

          Edit: just noticed what you said about the stdlib script. If you file an issue I can take a look at that this weekend.

          1. 2

            Thanks for the reply. I will give the latest version a try and see if it works with the stdlib. I am quite excited to see if it works now :)

            1. 2

              For those following along, we’re continuing this discussion here: https://github.com/mvdan/sh/issues/624

        3. 2

          Thanks so much @mvdan shfmt has been a great tool in easing the drudgery of formatting code when composing bash! Both shfmt and shellcheck have turned bash from on avoided language to one I enjoy and respect.