I feel like this is something like nushell is also trying to solve. I’ve not daily driven nushell, only experimented with it, and I’ve not touched powershell in some years, so I can’t give a definitive answer on it. Both feel better UX and repeatability from a programmatic standpoint.
ironically, i think that nushell on *nix systems is a harder sell than powershell on windows because of compat, despite shells being a much larger part of the way people typically work on *nix systems.
i tried using nushell as my daily driver, and pretty frequently ran into scripts that assumed $SHELL was going to be at least vaguely bash-compatible. this has been true on most *nix boxes for the past 20+ years and has led to things quietly working that really shouldn’t have been written.
OTOH cmd.exe is a much smaller part of the windows ecosystem and (at least, it seems to me) there is much less reliance on the “ambient default shell behaves like X”, so switching to the new thing mostly just requires learning.
(i ultimately dropped nushell for other reasons, but this was also over 30 releases ago so its changed a bit since then)
You’ve brought up several valid points, but I want to split this up a bit.
The reason using something other than bash (or something extremely close, like zsh) is painful is due to the number of tools that want to source variables into the environment. This is actually a completely tractable problem, and my default *nix shell (fish) has good tools for working with it. It’s certainly higher friction, but it’s not a big deal; there are tools that will run a bash script and then dump the environment out in fish syntax at the end so you can source it, and that works fine 95% of the time. The remaining 5% of the time almost always has a small fish script to handle the specific use-case. (E.g., for ssh-agent, there’s https://github.com/danhper/fish-ssh-agent. Hasn’t been updated since 2020, but that’s because it works and is stable; there’s nothing else I could imagine really doing here.) And you could always set whatever shell for interactive-only if you really want it (so that e.g. bash would remain the default $SHELL).
PowerShell on Windows actually has to do this, too, for what it’s worth. For example, the way you use the Windows SDK is to source a batch file called vcvarsall.bat (or several very similar variants). If you’re in PowerShell, you have to do the same hoisting trick I outlined above–but again, there are known, good ways to do this in PowerShell, to the point that it’s effectively a non-problem. And PowerShell, like fish, can do this trick on *nix, too.
Where I see Nushell fall down at the moment is three places. First, it’s just really damn slow. For example, sometimes, when I’m updating packages, I see something fly by in brew/scoop/zypper/whatever that I don’t know what it is. 'foo','bar','baz','quux'|%{scoop home $_} runs all but instantly in PowerShell. [foo bar baz] | each { scoop home $it } in Nushell can only iterate on about one item a second. But on top of that, Nushell has no job control, so if I want to start Firefox from the command line, I have to open a new tab/tmux window/what-have-you so I don’t lock my window. And third, it’s still churning enough that my scripts regularly break. And there are dozens of things like this.
I really want to like Nushell, and I’m keeping a really close eye on it, but, at the moment, running PowerShell as my daily shell on *nix is entirely doable (even if I don’t normally do it). Nushell…not so much.
There are all the software problems, some of which you’ve detailed, and then there’s IMO the even bigger problem - the human problem :)
UNIX users don’t just use, love, and build with the “everything is a stream of bytes” philosophy, it almost becomes baked into their DNA.
Have you ever tried to have a discussion about something like object pipelines or even worse yet, something like what AREXX or Apple’s Open Scripting Architecture used to offer to a hardcore UNIX denizen?
99 times out of 100 it REALLY doesn’t go well. There’s no malice involved, but the person on the other end can’t seem to conceptualize the idea that there are other modes with which applications, operating systems and desktops can interact.
As someone whose imagination was kindled very early on with this stuff, I’ve attempted this conversation more times than I care to count and have pretty much given up unless I know that the potential conversation partner has at least had some exposure to other ways of thinking about this.
I’d say it’s kind of sad, but I suspect it’s just the nature of the human condition.
I believe that, in the end, it all boils down to the fact that plain text streams are human readable and universal. You can opt-in to interpreting them as some other kind of a data structure using a specialized tool for that particular format, but you can’t really do it the other way around unless the transmission integrity is perfect and all formats are perfectly backward and forward compatible.
I would argue that scripts that don’t include a shebang at the top of them are more wrong than the shell that doesn’t really know any better what to do with them.
I don’t want to pollute this thread with my love for nushell, but I have high expectations for it, and I’ve previously maintained thousands-of-lines scripts that passed shell-check and were properly string safe. Something I think many people just avoid thinking about. (example, how do you build up an array of args to pass to a command, and then properly string quote them, without hitting the case where you have an empty array and output an empty string that inevitably evokes an error in whatever command you’re calling – in nushell this doesn’t matter, you just call let opts = ["foo", "bar xyz"]; echo $opts and the right thing happens)
I would argue that scripts that don’t include a shebang at the top of them are more wrong than the shell that doesn’t really know any better what to do with them.
Oh, the scripts absolutely are the problem. Unfortunately, that doesn’t mean that they don’t exist. Just another annoying papercut when trying out something new
My experience with FreeBSD defaulting to csh is that scripts aren’t really the problem. Sure, some of them hard-code bash in the wrong location, but most of them are easy to fix. The real problem is one-liners. A lot of things have ‘just run this command in your shell’ and all of those assume at least a POSIX shell and a lot assume a bash-compatible shell. FreeBSD’s /bin/sh has grown a few bash features in recent years to help with this.
FreeBSD’s /bin/sh has grown a few bash features in recent years to help with this.
Oh that’s interesting, didn’t know that but makes sense. I’ve noticed this with busybox ash too – it’s been growing bash features, and now has more than dash, which is a sibling in terms of code lineage.
A related issue is that C’s system() is defined to run /bin/sh. There is no shebang.
If /bin/sh happens to be /bin/bash, then people will start using bash features unconsciously …
Also, system() from C “leaks” into PHP, Python, and pretty much every other language. So now you have more bash leakage …
example, how do you build up an array of args to pass to a command, and then properly string quote them, without hitting the case where you have an empty array
FWIW my blog post is linked in hwayne’s post, and tells you how to do that !!! I didn’t know this before starting to write a bash-compatible shell :)
I’m very tempted. I tried it out way back when it first came out and checked back in recently on them and it’s amazing at how far the Nushell project has come.
I’ve probably spent two years of my dev career writing only powershell. It certainly wasn’t planned that way but there came a time where I looked back and couldn’t think of a time writing something in a different language. I don’t really like powershell, but I will say it’s miles better than bash scripting. Lightyears worse than Python though. I don’t remember all the issues I had with it (I remember error handling & propagation were bad) but one obvious thing it does wrong is the way it treats lists with a single element as that element object instead of a list with one element. Really messes you up if you write something expecting a list and get an object instead (or something like this, pipelines might be involved, I don’t totally remember). Also the current working directory of powershell itself and the current working directory used when you call a .NET library function are different.
Anyway the moral of the larger story is you have to proactively advocate for taking on interesting work or you’ll be shunted into deploying things or fixing things in a scripting language for a bunch of your career because nobody else wants to do it.
One thing I dislike about PowerShell is that it came out pretty late after bash, zsh, and lots of other shells were already stabilized and popular. Yet, PowerShell still didn’t manage to create a “normal” language, and .ps1 scripts are not better in the terms of readability than bash scripts. It’s like MS wasn’t aware that other shells have existed, has disregarded everything what’s wrong with those existing shells, and has created something newer, but also with a lot of old problems. Also it really baffles me that today we are excited that a shell can output some items in columns, where shells from 1988 (AmigaDOS + ARexx) could communicate via network by only shell scripting. A lot of things in technology really feels like a regression, but I guess one needs to be old enough to see it. I fear this will only get worse.
I like a lot of the ideas that PowerShell copied from Smalltalk but I still can’t quite forgive it for building a verb-noun convention decades after noun-verb was widely known to be better for discovery.
Yup, as far as I remember they borrowed [ $x -lt $y ] from shell, but they did it in an incompatible way !!!
If you’re going to be incompatible, why not just use (x < y) ?
It seems like a bizarre cargo culting, to the point of indicating a lack of understanding … Unix shell SYNTAX is not why people use shell!
Pretty much everyone dislikes the syntax, and they did in 1993 too, more than a decade before PowerShell appeared.
The terrible syntax can be mostly traced to implementation hacks. Like x=y is different than x = y because shell had words before it had variables, so they just “stuffed” an assignment into a word.
Likewise test 1 -lt 2 and [ 1 -lt 2 ] is just easier to implement because you skip writing a lexer, like Awk has.
I gave up on powershell long ago when I bumped into an issue where it showed it’s teeth: it was not deterministic. I mean, it worked the microsoft-y way: if there were an edge case, they assumed you want it in a way they decided.
Specific example (memories fading): I wanted to return a list of values inside a list (which has only one member) so it decided I don’t want nested lists and returned only one level deep list.
My takeaway was that when you need some type safety in a dynamically typed language, ps is not for you.
(Pls fix me if I don’t remember correctly)
It’s not a determinism issue, but rather an odd behavior where one-element lists devolve to just the item. So if you really want a one-item list returned, you need to wrap it in a second one-item list. I could get into why it works that way, and I agree it’s stupid, but it’s at least consistent.
I feel like Unix is moving extremely slowly but inexorably towards “everything emits a stream of NL JSON” as a solution to the problem of streaming objects.
The remarkable thing about Powershell is that it appears to have independently discovered the concept of pipes. The concept of pipes was described in The Bell Systems Technical Journal, Vol 57, No 6, July-August 1978 in an article written by S. R. Bourne. There is no mention of this reference in any Microsoft publication that I know of, so I infer that the concept was independently rediscovered around 2004-2005, and incredibly enough, has the same name as the one dating back around 25 years earlier.
This is a very bizarre interpretation … the creator of PowerShell USED Unix pipes for decades before he created PowerShell, just like millions of other people!!!
Once you use it, I would not call it “independent rediscovery” …
I’ve been meaning to look into converting the Windows mess we have in CHICKEN where we support Cygwin with bash, mingw+msys with bash or plain mingw with cmd.exe into only supporting mingw with Powershell (and perhaps still having cygwin). The current situation is also very confusing for users, because if you run the “wrong” makefile (say mingw+msys makefile from cmd.exe or the mingw makefile from msys) things just don’t work properly (rather than breaking immediately). Unfortunately, my willingness to put up with operating Windows (especially the shitty “modern IE” VMs that are time limited) as well as time to work on software beyond my day job is very limited so it’ll probably never happen.
One question though - is Powershell ubiquitous enough to simply assume it’s there on Windows? Or does it require one to install extra stuff? And what happens when you try to run Powershell stuff from cmd.exe?
PowerShell is part of the default install (core bits of Windows depend on it), but you need to be a bit careful about versions. Newer versions are available as optional installs and add new features.
is Powershell ubiquitous enough to simply assume it’s there on Windows?
On Win10 and above it’s simply there, but I’m not sure about the differences in versions. Also it was simply there since Win7 I think but I can’t verify this (I only have access to win10).
And what happens when you try to run Powershell stuff from cmd.exe?
From what I see, cmd will fire up the default handler for “ps1” files. If it’s notepad, it will fire up notepad. But I’ve changed the default handler for “ps1” files to be “powershell.exe”, and now it seems that ps1 scripts run under CMD the same way as under PSH. But that required a manual step to make.
I feel like this is something like nushell is also trying to solve. I’ve not daily driven nushell, only experimented with it, and I’ve not touched powershell in some years, so I can’t give a definitive answer on it. Both feel better UX and repeatability from a programmatic standpoint.
ironically, i think that nushell on *nix systems is a harder sell than powershell on windows because of compat, despite shells being a much larger part of the way people typically work on *nix systems.
i tried using nushell as my daily driver, and pretty frequently ran into scripts that assumed
$SHELLwas going to be at least vaguelybash-compatible. this has been true on most *nix boxes for the past 20+ years and has led to things quietly working that really shouldn’t have been written.OTOH
cmd.exeis a much smaller part of the windows ecosystem and (at least, it seems to me) there is much less reliance on the “ambient default shell behaves like X”, so switching to the new thing mostly just requires learning.(i ultimately dropped nushell for other reasons, but this was also over 30 releases ago so its changed a bit since then)
You’ve brought up several valid points, but I want to split this up a bit.
The reason using something other than bash (or something extremely close, like zsh) is painful is due to the number of tools that want to source variables into the environment. This is actually a completely tractable problem, and my default *nix shell (fish) has good tools for working with it. It’s certainly higher friction, but it’s not a big deal; there are tools that will run a bash script and then dump the environment out in fish syntax at the end so you can source it, and that works fine 95% of the time. The remaining 5% of the time almost always has a small fish script to handle the specific use-case. (E.g., for
ssh-agent, there’s https://github.com/danhper/fish-ssh-agent. Hasn’t been updated since 2020, but that’s because it works and is stable; there’s nothing else I could imagine really doing here.) And you could always set whatever shell for interactive-only if you really want it (so that e.g. bash would remain the default$SHELL).PowerShell on Windows actually has to do this, too, for what it’s worth. For example, the way you use the Windows SDK is to source a batch file called
vcvarsall.bat(or several very similar variants). If you’re in PowerShell, you have to do the same hoisting trick I outlined above–but again, there are known, good ways to do this in PowerShell, to the point that it’s effectively a non-problem. And PowerShell, like fish, can do this trick on *nix, too.Where I see Nushell fall down at the moment is three places. First, it’s just really damn slow. For example, sometimes, when I’m updating packages, I see something fly by in brew/scoop/zypper/whatever that I don’t know what it is.
'foo','bar','baz','quux'|%{scoop home $_}runs all but instantly in PowerShell.[foo bar baz] | each { scoop home $it }in Nushell can only iterate on about one item a second. But on top of that, Nushell has no job control, so if I want to start Firefox from the command line, I have to open a new tab/tmux window/what-have-you so I don’t lock my window. And third, it’s still churning enough that my scripts regularly break. And there are dozens of things like this.I really want to like Nushell, and I’m keeping a really close eye on it, but, at the moment, running PowerShell as my daily shell on *nix is entirely doable (even if I don’t normally do it). Nushell…not so much.
You’re absolutely right. It’s a VERY hard sell.
There are all the software problems, some of which you’ve detailed, and then there’s IMO the even bigger problem - the human problem :)
UNIX users don’t just use, love, and build with the “everything is a stream of bytes” philosophy, it almost becomes baked into their DNA.
Have you ever tried to have a discussion about something like object pipelines or even worse yet, something like what AREXX or Apple’s Open Scripting Architecture used to offer to a hardcore UNIX denizen?
99 times out of 100 it REALLY doesn’t go well. There’s no malice involved, but the person on the other end can’t seem to conceptualize the idea that there are other modes with which applications, operating systems and desktops can interact.
As someone whose imagination was kindled very early on with this stuff, I’ve attempted this conversation more times than I care to count and have pretty much given up unless I know that the potential conversation partner has at least had some exposure to other ways of thinking about this.
I’d say it’s kind of sad, but I suspect it’s just the nature of the human condition.
I believe that, in the end, it all boils down to the fact that plain text streams are human readable and universal. You can opt-in to interpreting them as some other kind of a data structure using a specialized tool for that particular format, but you can’t really do it the other way around unless the transmission integrity is perfect and all formats are perfectly backward and forward compatible.
I would argue that scripts that don’t include a shebang at the top of them are more wrong than the shell that doesn’t really know any better what to do with them.
I don’t want to pollute this thread with my love for nushell, but I have high expectations for it, and I’ve previously maintained thousands-of-lines scripts that passed shell-check and were properly string safe. Something I think many people just avoid thinking about. (example, how do you build up an array of args to pass to a command, and then properly string quote them, without hitting the case where you have an empty array and output an empty string that inevitably evokes an error in whatever command you’re calling – in nushell this doesn’t matter, you just call
let opts = ["foo", "bar xyz"]; echo $optsand the right thing happens)I’ll just leave a small example, so as to not go overboard here. I ought to compile my thoughts more fully, with more examples. But even something like this: https://github.com/colemickens/nixcfg/blob/02c00ef7a3e1e5dd83f84e6d4698cba905894fc7/.github/clean-actions.nu would not exactly be super fun to implement in bash.
Oh, the scripts absolutely are the problem. Unfortunately, that doesn’t mean that they don’t exist. Just another annoying papercut when trying out something new
My experience with FreeBSD defaulting to csh is that scripts aren’t really the problem. Sure, some of them hard-code bash in the wrong location, but most of them are easy to fix. The real problem is one-liners. A lot of things have ‘just run this command in your shell’ and all of those assume at least a POSIX shell and a lot assume a bash-compatible shell. FreeBSD’s /bin/sh has grown a few bash features in recent years to help with this.
Oh that’s interesting, didn’t know that but makes sense. I’ve noticed this with busybox ash too – it’s been growing bash features, and now has more than dash, which is a sibling in terms of code lineage.
A related issue is that C’s system() is defined to run /bin/sh. There is no shebang.
If /bin/sh happens to be /bin/bash, then people will start using bash features unconsciously …
Also, system() from C “leaks” into PHP, Python, and pretty much every other language. So now you have more bash leakage …
FWIW my blog post is linked in hwayne’s post, and tells you how to do that !!! I didn’t know this before starting to write a bash-compatible shell :)
Thirteen Incorrect Ways and Two Awkward Ways to Use Arrays
You don’t need any quoting, you just have to use an array.
The
--protects against filenames that look like flags.As mentioned at the end of the post, in YSH/Oil it’s just
Much easier to remember :)
This has worked for years, but we’re still translating it to C++ and making it fast.
Ah, yes, you certainly know what I (erroneously) was referencing (for others since I mis-explained it: https://stackoverflow.com/questions/31489519/omit-passing-an-empty-quoted-argument). Indeed most times starting with an empty array and building up is both reasonable and more appropriate for what’s being composed/expressed.
I’m very tempted. I tried it out way back when it first came out and checked back in recently on them and it’s amazing at how far the Nushell project has come.
I’ve probably spent two years of my dev career writing only powershell. It certainly wasn’t planned that way but there came a time where I looked back and couldn’t think of a time writing something in a different language. I don’t really like powershell, but I will say it’s miles better than bash scripting. Lightyears worse than Python though. I don’t remember all the issues I had with it (I remember error handling & propagation were bad) but one obvious thing it does wrong is the way it treats lists with a single element as that element object instead of a list with one element. Really messes you up if you write something expecting a list and get an object instead (or something like this, pipelines might be involved, I don’t totally remember). Also the current working directory of powershell itself and the current working directory used when you call a .NET library function are different.
Anyway the moral of the larger story is you have to proactively advocate for taking on interesting work or you’ll be shunted into deploying things or fixing things in a scripting language for a bunch of your career because nobody else wants to do it.
This is all very well and nice, and I’ve actually used PowerShell a lot on windows. But sometimes a man just wants to grep some shit.
In my experience, PS/Windows is really slow when accessing text files. I usually just use WSL when wanting to to unixy stuff to text files.
One thing I dislike about PowerShell is that it came out pretty late after bash, zsh, and lots of other shells were already stabilized and popular. Yet, PowerShell still didn’t manage to create a “normal” language, and
.ps1scripts are not better in the terms of readability than bash scripts. It’s like MS wasn’t aware that other shells have existed, has disregarded everything what’s wrong with those existing shells, and has created something newer, but also with a lot of old problems. Also it really baffles me that today we are excited that a shell can output some items in columns, where shells from 1988 (AmigaDOS + ARexx) could communicate via network by only shell scripting. A lot of things in technology really feels like a regression, but I guess one needs to be old enough to see it. I fear this will only get worse.I like a lot of the ideas that PowerShell copied from Smalltalk but I still can’t quite forgive it for building a verb-noun convention decades after noun-verb was widely known to be better for discovery.
Would you have a source for that? I like verb-noun better but am very ready to have my mind changed. Much obliged!
Yup, as far as I remember they borrowed
[ $x -lt $y ]from shell, but they did it in an incompatible way !!!If you’re going to be incompatible, why not just use
(x < y)?It seems like a bizarre cargo culting, to the point of indicating a lack of understanding … Unix shell SYNTAX is not why people use shell!
Pretty much everyone dislikes the syntax, and they did in 1993 too, more than a decade before PowerShell appeared.
The terrible syntax can be mostly traced to implementation hacks. Like
x=yis different thanx = ybecause shell had words before it had variables, so they just “stuffed” an assignment into a word.Likewise
test 1 -lt 2and[ 1 -lt 2 ]is just easier to implement because you skip writing a lexer, like Awk has.I gave up on powershell long ago when I bumped into an issue where it showed it’s teeth: it was not deterministic. I mean, it worked the microsoft-y way: if there were an edge case, they assumed you want it in a way they decided.
Specific example (memories fading): I wanted to return a list of values inside a list (which has only one member) so it decided I don’t want nested lists and returned only one level deep list. My takeaway was that when you need some type safety in a dynamically typed language, ps is not for you.
(Pls fix me if I don’t remember correctly)
It’s not a determinism issue, but rather an odd behavior where one-element lists devolve to just the item. So if you really want a one-item list returned, you need to wrap it in a second one-item list. I could get into why it works that way, and I agree it’s stupid, but it’s at least consistent.
What’s with the powershell color scheme though?
You can change the color scheme (from inside powershell even!) by assigning new values to the keys of
$host.ui.rawui.do you or anyone have a nice scheme ready to paste in?
For theming, use something like OhMyPosh.
Makes it very easy to style the terminal (and it’s also x-plat).
I feel like Unix is moving extremely slowly but inexorably towards “everything emits a stream of NL JSON” as a solution to the problem of streaming objects.
What’s NL? Does it mean that there is one JSON object for each NewLine?
Yeah, I’d heard it named ndjson, but same deal.
I very much love
cut, but in this case as the author points out, it is not the right tool. That being said,awkcan save the day:In my example
lsdoes not have the--full-timeoption, but in any caseawkworks nicely with formatted columns.The remarkable thing about Powershell is that it appears to have independently discovered the concept of pipes. The concept of pipes was described in The Bell Systems Technical Journal, Vol 57, No 6, July-August 1978 in an article written by S. R. Bourne. There is no mention of this reference in any Microsoft publication that I know of, so I infer that the concept was independently rediscovered around 2004-2005, and incredibly enough, has the same name as the one dating back around 25 years earlier.
This is a very bizarre interpretation … the creator of PowerShell USED Unix pipes for decades before he created PowerShell, just like millions of other people!!!
Once you use it, I would not call it “independent rediscovery” …
??? The founding document (Monad Manifesto) explicitly discusses Unix pipes.
Windows has had pipes for ages too. Powershell’s are just typed as object streams instead of byte streams.
Just so you don’t feel alone, I chuckled at the joke.
OK, probably was a joke … but making fun of Microsoft for borrowing good ideas? I don’t really get it
I would rather make fun of them for copying the
$x -lt $ysyntax, but in an incompatible way :)I’ve been meaning to look into converting the Windows mess we have in CHICKEN where we support Cygwin with bash, mingw+msys with bash or plain mingw with
cmd.exeinto only supporting mingw with Powershell (and perhaps still having cygwin). The current situation is also very confusing for users, because if you run the “wrong” makefile (say mingw+msys makefile fromcmd.exeor the mingw makefile from msys) things just don’t work properly (rather than breaking immediately). Unfortunately, my willingness to put up with operating Windows (especially the shitty “modern IE” VMs that are time limited) as well as time to work on software beyond my day job is very limited so it’ll probably never happen.One question though - is Powershell ubiquitous enough to simply assume it’s there on Windows? Or does it require one to install extra stuff? And what happens when you try to run Powershell stuff from
cmd.exe?PowerShell is part of the default install (core bits of Windows depend on it), but you need to be a bit careful about versions. Newer versions are available as optional installs and add new features.
On Win10 and above it’s simply there, but I’m not sure about the differences in versions. Also it was simply there since Win7 I think but I can’t verify this (I only have access to win10).
From what I see, cmd will fire up the default handler for “ps1” files. If it’s notepad, it will fire up notepad. But I’ve changed the default handler for “ps1” files to be “powershell.exe”, and now it seems that ps1 scripts run under CMD the same way as under PSH. But that required a manual step to make.