1. 11

My first thoughts on how to ensure POSIX shell compatibility and making sure no new bashisms are introduced in the future are: test on different POSIX shells using CI and use a mix of different POSIX shells, i.e. OpenBSD sh and Debian dash. I wonder what others think of this issue and how to approach a possible solution?

I’m also wondering if anyone knows of a complete and free POSIX shell implementation? I know OpenBSD sh and dash have differences for example and I’m not sure they’re 100% compliant.

I’m totally new to Go so I first wanted to consult my fellow Lobsters before discussing further in the Go thread.

EDIT: for clarity, I don’t know the person who started that thread on GitHub but it was created right before I was wondering this myself and I’m contemplating whether or not I would like to do this work as a newcomer to Go, whether it would be appreciated, and get a feel of what a possible solution would look like.

  1.  

  2. 19

    Forgive me if this is gauche, but what is wrong with simply using bashisms? Outside of embedded contexts, where you want everything in Busybox, but writing shell scripts in just POSIX shell just seems like a tortured dialect.. The extensions are legitimately useful, so it’s also a question of why other shells haven’t implemented it.

    Also curious is not wanting to use Shellcheck, even if it’s just for (skippable) CI-side tests. Shellcheck was the first tool that made writing shell scripts tolerable for me.

    1. 4

      Some POSIX operating systems don’t come with Bash out of the box, notably the BSDs. As such Bash is rarely used in them even if it is available. Even MacOS switched its default shell to ZSH.

      Generally though I think dropping the dependency on Bash increases compatibility across the board and removes an unneeded dependency. Both of which are always welcome.

      1. 3

        I don’t mind taking dependencies if it helps you reduce complexity elsewhere, especially if the cost is amortized elsewhere.

        1. 4

          You’re not the one maintaining thousands of rc scripts or build scripts for a distro/flavor. Or at least I assume you’re not. The tradeoffs communities make usually have a reason and just because you don’t see it or understand it doesn’t mean it’s not there.

          1. 3

            But this is not about thousands of rc scripts. This issue is about one script used during the go build process. On majority of systems it will need a dependency that is immediately satisfied. On some minority it will require a single package.

            1. 4

              The GitHub issue appears to be a troll issue, and I agree it doesn’t really matter much in the context of the Golang toolchain. However I was responding to the thread which was speaking more generically about dependencies and script maintenance.

              1. 3

                Well technically at least 3.

                But I agree in general. I fail to see why requiring bash is such a huge deal (I have read the comments here as well as the comments on the GitHub issue).

          2. 2

            Even MacOS switched its default shell to ZSH.

            For a while, not any more.

            1. 1

              What is it now?

              1. 1

                I’m fairly sure it’s back to bash.

          3. 3

            One of the comments from that issue:

            Just had my first experience with Go, which was nice. The one thing that surprised me a little bit was that bash was required to build. On OpenBSD the standard shell is a hardened ksh. bash is avoided everywhere in base so I had to install that, no biggie, but the question I would phrase in the spirit of portability and reducing dependencies is “why require more if technically all you need is POSIX shell”? I’m wondering if such a change would be desirable, aside from the question who’s going to make it happen.

            Also, doesn’t macOS uses zsh by default now? But probably, it’ll be compatible with the bash scripts used here?

            1. 3

              It seems like just another dependency to me though - and certainly one common enough most people will have, and run on most people’s systems.

              1. 3

                doesn’t macOS uses zsh by default now?

                Yes, but they’re not removing bash from the standard system AFAIK

                1. 2

                  Yes, that comment is from me. It is my first encounter with the Go community, so after discovering they don’t completely dismiss the idea itself, I thought let’s first get some broader opinions and have a discussion with a community I’m part of before I continue this discussion in the Go thread.

                  Also, doesn’t macOS uses zsh by default now?

                  idd

                  But probably, it’ll be compatible with the bash scripts used here?

                  Good question, from a superquick check I can say it doesn’t right out fail like it does on OpenBSD with ksh.

                2. 2

                  Forgive me if this is gauche, but what is wrong with simply using bashisms?

                  Nothing, bash really makes our lives easier, the extensions are really useful. Some people wants to only use POSIX sh, just because it is a “standard”. Some people just hates bash because it is popular.

                3. 7

                  There is 100% compliant vs 100% restricted to what is stated in POSIX. As far as I know the latter does not exist, and it cannot because there isn’t enough specified by POSIX, at some point you need to take some decisions on what is not specified and other implementations might take the opposite direction.

                  For example, how do you decide if “echo foo | read var” should fork two processes or only one? After this line the value of var is different between bash and zsh because zsh decides to not fork the last member of the pipeline.

                  Even though there’s nothing outside of the scope of POSIX here, we still have a completely different result.

                  Sure with shellcheck you can avoid most of those pitfalls, but if your goal is to ensure portability, I think it is much easier to only allow a specific shell to run your script, even if it creates a dependency.

                  1. 7

                    I had the same knee-jerk reaction :D - at the time I was on a “porting” roll, having just converted the git-prompt stuff to OpenBSD’s ksh.

                    After further reflection, it became obvious that converting the build system (wrapper?) would potentially introduce more issues than it solves. Sorta a “if it ain’t broke” situation..

                    If you are looking specifically for Go things to help with this label has a lot of stuff that one can take a crack at!

                    If you are looking for OpenBSD+Go things - There is a grip of that too! I have documented a few things here. IMO enabling PIE mode on OpenBSD would be a decent start - it gets ya into various bits in the Go runtime - and eventually into some OpenBSD areas (that I haven’t been able to track down the breakage on :D).

                    I also know that jsing@ is looking for some help switching things from using syscalls to using libc. That change would let OpenBSD remove the Go specific loosening in the kernel!

                    1. 2

                      I had the same knee-jerk reaction :D

                      Well, the “knee-jerk” reaction is to the person who started that thread for not coming up with further details. I found the reaction of ianlancetaylor to my particular comment very helpful, at least it gives me the idea that if someone wants to step up and make this happen, there is fair chance it will be included, with the caveat on how to prevent backsliding to bashisms, hence the discussion I started here on Lobste.rs.

                      After further reflection, it became obvious that converting the build system (wrapper?) would potentially introduce more issues than it solves. Sorta a “if it ain’t broke” situation..

                      Thanks for sharing that :) I’m a bit afraid / hesistant for that as well, as most people are I guess.

                      Thanks for the other pointers as well! The whole reason I was building the runtime myself is because while pledging an spf filter I found that only LookupHost and LookupAddr can be handled by libc (and call get{addr,name}info), but other lookups, i.e. LookupTXT always go through native Go, hence I had to pledge “inet” instead of only “dns”. So another thing I’m thinking of is making sure that more of the Name Resolution is handled via libc using res_init(3) so that code that only needs dns from the network only needs a “dns” pledge instead of the full “inet”.

                    2. 5

                      I’m also wondering if anyone knows of a complete and free POSIX shell implementation?

                      There is a work-in-progress shell by emersion aiming to be strictly POSIX compliant and no more: https://git.sr.ht/~emersion/mrsh

                      1. 5

                        Speaking purely to the contribution angle: this wouldn’t be a very good first project. It’s not a good-faith request, it doesn’t actually solve a problem, and even if you did the work it’s unlikely to be accepted.

                        1. 3

                          I packaged go for my own package manager, it requires that go depends on bash to build, which increases build times by a fair amount.

                          It is a real problem for me anyway.

                          1. 1

                            Is make.bash the only place where Go depends on bash?

                            1. 1

                              I didn’t run the test suite (Integrating test suites with the package manager is a WIP) so not totally sure. It seems to work without it though.

                              I would agree its not a major, or really important problem, I was just trying to make the point that unnecessary dependencies are at least a small annoyance. Shrugging them off as no problem is not something I agree with.

                        2. 4

                          If you’re concerned with POSIX compliance, use dash. It’s (unverified) the most compliant.

                          If you’re concerned with getting things done on most (not embedded, not boot or recovery) systems you encounter just use bash explicitly. I recommend using shellcheck too.

                          As far as that ticket? I know that individual and I’ll leave it at that.

                          1. 2

                            If you’re concerned with POSIX compliance, use dash. It’s (unverified) the most compliant.

                            Mmm, I’m just reading: dash is actually the most “divergent” shell I’ve tested, because of stuff the spec leaves open for implementors.

                            /edit punctuation

                            1. 1

                              Surprised that there is undefined behavior in a shell? The matter is does your particular script fail or behave oddly where you need it to run. Otherwise this is a bit of navel-gazing.

                              My opinion is that POSIX compliance isn’t what we believe/hope it is. What I usually want is portability and predictability for the systems I interact with. On Linux that’s bash in Bourne sh mode.

                          2. 4

                            I’m also curious if @andyc has any thoughts on this as a shell developer.

                            1. 3

                              Yes thanks, I generally don’t believe in “POSIX minimalism” for the reasons you quoted, e.g.

                              https://lobste.rs/s/1tcqoo/introduction_posix_shell#c_jebkjl

                              Looking at the patch

                              https://github.com/kisslinux/community/blob/d99595e575a471698032dbad0cd2c9977d6b0c79/community/go/patches/posix-build.patch

                              As far as I can tell this is one of the “short scripts” which could reasonably be POSIX.

                              But I understand that the maintainers have better things to do with their time, and there is no way to prevent regression.

                              That is what I meant in my FAQ that POSIX is an “untestable” concept. The real goal is portability, and that means people should be running on multiple shells. If lots of people are doing that it makes sense to accept the patch.

                              But again OpenBSD ksh accepts A LOT OF STUFF that’s not POSIX. So I think that for example, the intersection of bash + ksh makes a lot more sense as a portability goal. (Flip side: I suppose that OpenBSD has been running bash for decades, so there’s really no problem here to justify the portability.)

                              Related comment: POSIX shell misses a lot of portable constructs


                              For bigger scripts, my advice is here:

                              https://news.ycombinator.com/item?id=24523162

                              If you want a portable shell script, in many cases my (biased) advice would be to make your script work on both bash and Oil. (Obviously there are short shell scripts which you may want to run on BSD, etc. This is more about big scripts, which POSIX falls down for.)

                              Oil already runs some of the biggest shell scripts in the world, many of them unmodified. Moreoever, when there’s a patch necessary to run it, it often IMPROVES the program.

                              http://www.oilshell.org/blog/2020/06/release-0.8.pre6.html#patch-to-run-the-mal-lisp

                              You’ll be less tied to the vagaries of bash.

                              If anyone’s script doesn’t run under Oil, I’m interested. See https://github.com/oilshell/oil/wiki/What-Is-Expected-to-Run-Under-OSH

                              1. 2

                                there is no way to prevent regression.

                                This is kind of bullshit. They already have CI scripts, just run it with more than one shell to check it still works. Run it with busybox, run it with dash, run it with ksh on the openbsd builds.

                            2. 4

                              I tend towards POSIX sh personally, and I do get mildly irritated when I find bashisms in #!/bin/sh scripts. But, this looks fine to me. I like that they’ve specifically identified it with a .bash extension, and the shebang is #!/usr/bin/env bash rather than #!/bin/bash (I’m just guessing that the latter might be an issue where bash isn’t in the base system). As there’s (I guess?) only one bash implementation there’s probably far less potential for running into incompatibilities. All in all, as someone with something of an aversion to bash, this usage feels perfectly reasonable.

                              1. 4

                                This is the background to the current situation

                                https://github.com/golang/go/issues/3235#issuecomment-66064352

                                In general, I don’t see any harm in using the POSIX shell as a baseline for new projects. I haven’t seen any compelling reasons here or in the linked GH issue for making such a change for the golang project now, other than implicitly ideological.

                                1. Golang is not part of the base of any BSD system, so bash can just be included as a prerequisite for a port.
                                2. While bash might be heavyweight, it’s well understood and documented. ‘bashisms’ are just another word for ‘convenient features in a Unix shell’. Golang itself requires significant resources to build.
                                3. this is a non-issue if golang is distributed as a binary, which I suspect it is for 99.9% of all users. Requiring a rework of the build system, with attending updates to CI setups, to satisfy a tiny minority who already have a workaround, is not a good use of engineering resources.
                                1. 3

                                  I like bash and use it daily, even though I know there are other more fully-featured bourne-ish shells out there.

                                  However, when I have a trivial script to write, or something that I think will have to run on multiple operating systems, I try to write in /bin/sh in an effort to be kind to my future self and others who might have to maintain what I wrote.

                                  Part of a cross-platform build system is exactly the right place to be using /bin/sh. If any of the build script would materially benefit from bashisms, it would imply to me that it is something best implemented in the “main” build system logic (make, or whatever Go uses).

                                  1. 2

                                    I’m genuinely curious about this approach. Isn’t bash ported and running on so many systems, that it is practically unavoidable? In addition, isn’t its package quite small? In other words, that is it that one can gain in writing something in /bin/sh, that is sometimes implemented differently in different OSes?

                                    1. 2

                                      Bash is very portable, yes, but it isn’t installed by default on all unix-likes and derivatives, especially BSDs (like OpenBSD) commercial Unices (like AIX) or specialized Linux distributions that might only have busybox for example. These days, /bin/sh is the lowest common denominator Unix shell. (Although if you go back far enough, that wasn’t always the case.)

                                      Bash is lightweight by today’s standards but is still an order of magnitude larger than, say, dash, even before you consider that bash pulls in a couple additional dynamically linked libraries as well:

                                      $ ls -l /bin/bash
                                      -rwxr-xr-x 1 root root 1,183,448 Jun 18 11:44 /bin/bash*
                                      $ ls -l /bin/dash
                                      -rwxr-xr-x 1 root root 129,816 Jul 18  2019 /bin/dash*
                                      

                                      For trivial scripts, interactive use, and one-off things, the difference is usually completely negligible but Debian and Ubuntu switched to dash as the default system shell more than a decade ago because doing so provided a relatively cheap speedup on boot and other times shell scripts are called often, such as cron jobs: https://lists.debian.org/debian-release/2007/07/msg00027.html

                                  2. 4

                                    Two make dialects and three to four underlying shell implementations each with a dialect split makes for fragile, hard to debug builds.

                                    I am not a big user of Go, but from mystandpoint the dev ecosystem strives for simplicity, tractability and the ability to fix problems in the entire stack regardless of where they occur. Given those philosophies, Go should remove things that cause if-statements and that are also outside of their direct control.

                                    1. 3

                                      I advocate not doing it.

                                      Upside: you’ll be able to use ksh for this one thing instead of bash.

                                      Downside: making a change like this will most likely introduce bugs that cause build failures. This will annoy strangers and expend some of their time. Even in the unlikely event that everything goes perfectly, maintainers’ time will be expended reviewing & testing.

                                      The upside is not interesting. The downside is noticeably bad. Don’t “fix” things that aren’t broken.