1. 28
    1. 27

      This article implicitly conflates the interactive shell with the shell that you use for scripting. On Ubuntu, /bin/sh is dash, which is lightweight and starts quickly, whereas the default interactive shell is bash. On FreeBSD, /bin/sh is a POSIX shell (and some people replace it with /rescue/sh, which is statically linked and starts even faster) and most people install something like zsh or bash as their interactive shell.

      There’s no reason not to have a fast-launching shell for scripting and a slower one for interactive use.

      1. 10

        Worth noting that dash is short for Debian Almquist Shell; FreeBSD’s sh is also a descendent of Kenneth Almquist’s shell. It was written as part of Keith Bostik’s campaign to get rid of AT&T code from BSD, to create the almost-entirely unencumbered 4.3BSD Net/2 distribution. A lot of this effort happened on comp.sources.unix on Usenet. There was some overlap between the BSD and GNU rewrite efforts; flex is prominent example, and there is plenty of BSD code in glibc.

        1. 5

          Agreed, but with the caveat that many bash scripts are not posix compliant because they use bash-specific features. These features are actually useful.

          So, I think there is some reason to want these slower-launching shells for scripting as well.

          1. 2

            FWIW, ksh93 is significantly faster than bash and has even more cool features. It can even use proper regexes in glob patterns —by pretending ~(E)— in addition to extended bash style extglob patterns (which bash borrowed from ksh93 in the first place).

            Unfortunately it’s installed by default basically nowhere and if you need to install an interpreter along with your script you might as well use a proper language.

          2. 5

            Another caveat is that even if your only shell is bash, it will likely start faster non-interactively than interactively, because it loads different files.

            There is a rat’s nest of startup files, and there are at least 3 cases.

            https://zwischenzugs.com/2019/02/27/bash-startup-explained/

            https://blog.flowblok.id.au/2013-02/shell-startup-scripts.html

            I think sh -i -c is a decent approximation of interactive shell startup time though. But that’s the one that matters less because you start interactive shells much less often than tools like Make or Docker start non-interactive shells.

            On the other hand, the rat’s nest means that it’s hard to optimize and test. I’ve tried to move things around in the past, and it’s pretty easy to break things, so now I just avoid changing my startup files too much. And also I really dislike when tools like package managers mutate those files.

            1. 2

              and most people install something like zsh or bash as their interactive shell.

              In my experience, some 5% or so install other shell. Most don’t even know what that is and conflate shells with terminal emulators.

              In almost all scripts I see, people put bash in the shebang line instead of bourne shell.

              1. 2

                I very, very much doubt that more than a token number of FreeBSD users are putting bash on the shebang line.

              2. 1

                Agreed. I wrote down some similar thoughts after reading this article https://hermanradtke.com/developer-experience-fast-startup-is-not-the-only-speed-metric/

              3. 8

                Why not just add shebang lines to your shell scripts?

                1. 1

                  The use case here would be executing a script vs being on a logon shell. The shebang won’t help here other than existing scripts you write yourself. What the author writes is much more helpful if you have a heavy interactive shell setup.

                  1. 5

                    What scripts are you running that don’t include shebang lines? This seems like a convoluted and fragile solution to the problem of efficiently running shell scripts that were written by someone who didn’t even care enough to specify an interpreter.

                    1. 3

                      I noted that shebang won’t help, not that there isn’t one. Though a shebang on your shellrc sounds extremely silly and useless, possibly even breaking.

                      Your interpreter doesn’t matter for bashrc/bash_login/profile setup. Those script will run, in case of _login and profile they will run almost everytime the shell is initialized, for example when accessing a host via SSH. Things like ansible will be slowed down because they have to get past your shell before they can poke around the host and see if they can invoke python directly. And if you run a shell script it’ll all be loaded anyway.

                      There is also the practise of using $SHELL to invoke subscripts using the current shell, but it will ignore the shebang and means you’re running a new copy of your shell with all the heavy shellrc machinery.

                      1. 1

                        Okay. That makes more sense. I was thinking only in the local context, not executing stuff remotely via ssh. When I log into an interactive session I don’t really care about the one time startup cost of my interactive shell, but I can see how that could add up over repeated remote logins to run individual commands.

                2. 7

                  I think needing to profile your shell is mostly a zsh thing. Sure some of the external binaries add time, but interactive shell startup is just much easier to bog down with zsh. I spent years and tried many of the different optimizations, but it’s an ongoing battle.

                  I switched to fish a few years ago and haven’t looked back. Worrying about startup time is now a thing of my past.

                  1. 7

                    Not so much a zsh thing, as much as an oh-my-zsh thing. A several years ago, I had startup time issues, but all of them could be traced to bloat I was not using in OMZ. I have all the completions, git integration, etc that I could ever need in 130 lines of zshrc, and my startup times are exactly the same as OP.

                    oh-my-zsh was my introduction to zsh, and I’m grateful for that. But once I figured out which features were useful to me, all it took was a little effort to pare down to the ones I needed.

                    1. 1

                      I ran the command on my fish shell and wondered what the deal is.

                    2. 6

                      Based on the title, I thought this was gonna be about maintaining a pool of shells, so that you always had one warmed up when you open a new window/tab/split.

                      1. 5

                        Just what are you people putting in your interactive shell startups to make all this worthwhile?

                        1. 5
                          openbsd$ time ksh -i -c exit
                              0m00.00s real     0m00.00s user     0m00.01s system
                          

                          this article doesn’t even mention pdksh, but it smokes all the other examples

                          1. 3

                            And while it’s not exactly a fair comparison (fork/exec on 9front is also significantly faster than unix, shells aside), I had to try out the 9front rc shell; I had to put in a loop to get it to register:

                            % time rc -c 'for(i in `{seq 1000}) rc -i -c exit'
                            0.14u 0.41s 0.55r 	 rc -c for(i in `{seq 1000}) rc -i -c exit
                            
                            1. 2

                              Same goes for mksh :

                              $ time mksh -i -c exit
                                  0m00.00s real     0m00.00s user     0m00.00s system
                              
                              $ time sh -c 'for i in $(seq 1000); do mksh -i -c exit; done'
                                  0m01.53s real     0m01.17s user     0m00.30s system
                              
                            2. 4

                              An obvious question is: which shell is used to run sub-commands? Surprisingly, there is no simple answer to this, and different programs use different shells to run sub-commands. Some use the contents of the $SHELL environment variable; some use the shell set in the user database (i.e. /etc/passwd and friends); and some use a hard-coded shell (e.g. /bin/sh). In most cases, $SHELL and the shell in the user database are the same, often a “big” shell like bash, fish, or zsh. In many of the cases I care about, my non-interactive commands are run with one of these big shells, so the overhead of them is something I care about.

                              I don’t feel like this is accurate ? What kind of Unix system are you using?

                              • On say Debian, every script has a shebang line, usually #!/bin/sh
                              • Scripts in tarballs and git repos will often have #!/bin/bash or similar
                              • If there’s no shebang line, the shell will use /bin/sh (i.e. shells invoking shell scripts without shebangs have a special rule)
                              • The system() call in libc is defined by POSIX to use /bin/sh. So if you do os.system() in Python, it will use /bin/sh.
                              • GNU make will use /bin/sh unless you override it with $SHELL I believe.
                              • Ninja parses the command lines and makes its own argv array; it doesn’t need to use a shell at all.

                              On say Debian I’ve never heard of any case where the kind optimization you’re suggesting matters. But maybe that’s because Debian switch /bin/sh to dash a long time ago.

                              Though I seem to remember that they actually couldn’t measure the performance benefit from switching /bin/sh from bash to dash … there were some people upset about that. The main thing was that bash was viewed as sort of bloated and unreliable, and badly documented.


                              I guess to be more specific, my shell on Debian is /bin/bash according to /etc/passwd.

                              But I don’t anticipate any speedup at all by switching it to /bin/sh, even though dash (which is /bin/sh) is slightly faster than bash – and honestly I’m not sure that’s even true in the non-interactive case! They both seem to take 1ms or less.

                              Where would switching to /bin/sh make a difference? Most things that need to be fast already use /bin/sh without me configuring it – on Debian, but I think that’s true on all Unixes, just because of the way shebang lines and system() work.

                              1. 2

                                ssh <host> <cmd> is a widespread example of something that uses the shell set in the user database.

                                GNU make will use /bin/sh unless you override it with $SHELL I believe.

                                FWIW, I think it ignores $SHELL in the environment, but if you set SHELL it in the Makefile, it is used.

                                1. 1

                                  Hm, ssh respects the interactive / non-interactive distinction too, so the interactive startup files shouldn’t run when you supply <cmd>.

                                  % ssh oilshell.org 'echo $-'    
                                  hBc
                                  
                                  % ssh oilshell.org          
                                   $ echo $-
                                  himBHs
                                  

                                  Also, IME ssh itself is usually an order of magnitude slower than shell startup.

                                  I don’t doubt you sped SOMETHING up, I’m just not seeing from the post what exactly was slow, and what the fix was.

                                  I don’t think chsh /bin/sh is good general advice – seems like a good way to break a lot of stuff you want to work, at least on Debian! dash is a very limited shell compared to bash. bash is pretty fast these days, as configured on Debian.

                                2. 2

                                  If there’s no shebang line, the shell will use /bin/sh (i.e. shells invoking shell scripts without shebangs have a special rule)

                                  No, it uses the current shell:

                                  ❯ echo "$SHELL"
                                  /run/current-system/sw/bin/bash
                                  ❯ cat test.sh 
                                  echo "$SHELL"
                                  ❯ ./test.sh 
                                  /run/current-system/sw/bin/bash
                                  

                                  GNU make will use /bin/sh unless you override it with $SHELL I believe.

                                  That’s almost correct; it seems it uses sh and assumes that’s on $PATH:

                                  ❯ cat Makefile 
                                  test:
                                  	@echo $(SHELL)
                                  ❯ nix-shell -p gnumake
                                  ❯ make
                                  sh
                                  
                                  1. 1

                                    It depends on the shell! Looks like bash will start interpreting the shebang-less file itself, but dash execs /bin/sh.

                                    [pid 30450] execve("./z", ["./z"], 0x558ce58eab20 /* 27 vars */) = -1 ENOEXEC (Exec format error)
                                    [pid 30450] openat(AT_FDCWD, "./z", O_RDONLY|O_NOCTTY) = 3
                                    [pid 30450] read(3, "echo hi\n", 128)   = 8
                                    [pid 30450] execve("/bin/sh", ["/bin/sh", "./z"], 0x558ce58eab20 /* 27 vars */) = 0
                                    

                                    The more reliable test is to strace, since $SHELL can be exported and wrong.

                                    $ dash -c 'echo $SHELL'
                                    /bin/bash
                                    
                                3. 4

                                  My zsh starts in 14ms :)

                                  I spent a bunch of time back in 2015 to make it fast: https://github.com/stapelberg/configfiles/commits/master/zshrc?after=8cbbf81da4ea02cb3de551f150d34f34cfda14c0+69 contains the commits (Nov 11 + Nov 12), and it was definitely worth it!

                                  1. 4

                                    Fish shell with a lean config, ~13ms outside of a git repo, ~15ms inside a big git repo (where I get a prompt with the status of the repo).

                                    I removed conda initialization stuff from the config some time ago because it slowed it down significantly.

                                    1. 1

                                      This article made me remove conda as well. It was adding 170 ms to zsh startup time. What the heck is it doing??

                                    2. 4

                                      This article is a good prompt to take a gander at your shell rc file, which is always worth doing from time to time - I solved this problem another way though: caching! I keep a staged shell process in the background, and when I start a terminal what I’m actually doing is picking up the staged process, so the startup time is 0 (aside from window creation).

                                      1. 3

                                        Ohh this sounds interesting, how do you achieve this? I tried a quick search but nothing useful shows up

                                        1. 2

                                          I cheat! I do it in emacs, because it is my shell: https://github.com/neeasade/emacs.d/blob/e998595592ac35dd3195e44360abe4bd3b148751/lisp/trees/shell.el#L127-L197

                                          But, if you wanted to create this with a regular terminal-shell flow, I would reach for reptyr. The baby steps are there, but the interactive bash process seems to “pause” when it’s disowned

                                          $ bash -i & disown 
                                          [1] 1245161
                                          
                                          $ kitty reptyr 1245161
                                          

                                          another option could be creating a terminal window and hiding it with your WM, and then have “spawn terminal” = “unhide staged terminal window, fork off restaging”

                                          1. 2

                                            this seems interesting! are you also one of the emacs OS person :P

                                            reptyr seems like an interesting project! Tho I don’t think I have long running processes around a lot, but will keep in mind!

                                            As for the shell startups, if I use reptyr, I think I will have to make my own reptyr managment script? Which detects if the terminal under it is used, if yes, it spawns a new one. Maybe a script at the start of bashrc, which tries to query from retpyr management tool?

                                            I just realized we were doing it to reduce shell startup times and I added one more script to bashrc :sob:

                                            1. 2

                                              are you also one of the emacs OS person :P

                                              very much so!

                                              As for the shell startups, if I use reptyr, I think I will have to make my own reptyr managment script? Which detects if the terminal under it is used, if yes, it spawns a new one. Maybe a script at the start of bashrc, which tries to query from retpyr management tool?

                                              I just realized we were doing it to reduce shell startup times and I added one more script to bashrc :sob:

                                              all very true lmao, just the name of the game, and what hacks you are comfortable carrying around (everyone’s dots are broken in specific ways just for them✨)

                                      2. 3

                                        It’s hard enough getting scripts to work in a single version of a single Unix shell; keeping extremely subtle shell differences in short-term memory seems like a waste.

                                        1. 2

                                          I don’t think the non-interactive shell ends up being POSIX-ly correct. It is not strictly /bin/sh, because RegExp matching with [[ is a Dash/Bash/Zsh/etc. feature.

                                          1. 2

                                            Debian’s /bin/sh will just always fail on the test since [[ isn’t found, so you’ll always end up with /bin/sh.

                                            1. 2

                                              Is there a good alternative? Just plain grep I suppose?

                                              1. 3

                                                You could use a case statement, glob matching would work there, or you could just skip the test entirely since ~/.profile is only sourced for login shells.

                                                1. 2

                                                  case is an excellent idea – I’ve updated the post accordingly!

                                            2. 2

                                              I was glad to see someone commented on my favorite shell fish! As a philosophical aside , how come so many people like ZSH? I haven’t really bothered with it because fish kind of does everything that I need.

                                              1. 2

                                                According to hyperfine, my bash takes ~13ms, with the fancy prompt.

                                                I wrote the prompt in Zig so that it’s easy to ensure that there’s no allocations, only one write call, etc. But I never really measured it like this. Seems okay, considering all the other stuff that’s in there.

                                                1. 1

                                                  On the work laptop, which is a bit newer, I get 6.8ms.

                                                2. 2

                                                  For a hot-cache time, I get

                                                  $ (repeat 20 ZSH_TIME_STARTUP= zsh -lic exit)|& cstats -m3
                                                  0.025019 +- 0.000023 seconds
                                                  

                                                  But I do zcompile my zcompdump file after profiling revealed that as a major source of time.

                                                  EDIT: But in non-interactive use with an empty environment, statically linked dash starts up in only about 210 +- 16 microseconds. So, you know 125X more for all that Zsh interactive sugar (which I do love!).

                                                  1. 1

                                                    cstats

                                                    I wonder where this is coming from? :)

                                                    1. 2

                                                      https://github.com/c-blake/bu/blob/main/cstats.nim but, more portably phrased, it’s just the {mean+-stderr(mean)}(min-3 points). Of course the whole distribution (aka vulnerability to depths all over the many queues in any modern CPU/OS) can also be important, but, since there is a true, shared hard minimum time, the most happy path is the most reproducible. Understanding (usually) starts with reproducibility { & people also sure love their single-summary-hot cache times :) }.

                                                      1. 2

                                                        Thanks!

                                                  2. 1

                                                    I really like having information about git status in my prompt (if I’m in a git repository), but that can slow down “time to next prompt” to a noticeable degree (especially in large git repos). The linked article recommends removing the information from your prompt and just running commands (e.g., git status) in the shell. That will definitely fix the delay, but as an alternative I recommend that people look for async solutions. I’m currently using woefe/git-prompt.zsh to display git status in zsh, and it completely removes the drag from git lookups. (I’ve also seen solutions for zsh that use a daemon or a cache, but I haven’t investigated those.)

                                                    Also, again for zsh, a good tool for timing how long it takes to get a new prompt on zsh is zsh-prompt-benchmark. It can be useful if you are tweaking things and want to check for improvement.

                                                    1. 1

                                                      Nushell for me on a low-end VPS (HDD, 2 vCores) takes about 40ms. That’s also with Starship enabled.

                                                      1. 1

                                                        Yayy.

                                                        Both phone and my NixOS computers are under 100ms. Laptop around 30ms, Termux at about 50ms.

                                                        I use bash, custom vanilla PS1 (\u@\h \w $?\$ \[$(tput sgr0)\])

                                                        1. 1

                                                          Bash takes less than 40 ms on my laptop with things like Starship, direnv, and shared history between terminals configured. Not bad!

                                                          A major factor is probably that all those tools which want to be loaded in every Bash shell are instead loaded only in the relevant shell.nix.

                                                          1. 1

                                                            Shaved 20ms of mine, from:

                                                            $ hyperfine -N -w 10 --min-runs 500 'bash -i -c exit'
                                                            Benchmark 1: bash -i -c exit
                                                              Time (mean ± σ):      41.9 ms ±  45.2 ms    [User: 29.3 ms, System: 10.6 ms]
                                                              Range (min … max):    19.3 ms … 600.1 ms    500 runs
                                                            

                                                            to

                                                            $ hyperfine -N -w 10 --min-runs 500 'bash -i -c exit'
                                                            Benchmark 1: bash -i -c exit
                                                              Time (mean ± σ):      22.0 ms ±  22.9 ms    [User: 15.3 ms, System: 4.0 ms]
                                                              Range (min … max):    12.8 ms … 254.0 ms    500 runs
                                                            

                                                            Maybe I can enhance it a bit, but that’s not 1s neither :)