This rules. PowerShell had the right idea and executed on the Unix philosophy (small, composable programs which do one thing well) but hitched its horse to an overly-verbose C# API, awful names, awful error messages, and so on. It’s extremely good to see the same idea executed with what looks like a really nuanced understanding of where PowerShell comes up short in interacting with real data / external tools.
Having used and developed for PowerShell seriously for several years, “look similar to sh” is not enough, particularly when small incompatibilities and naming conflicts impede interop with common Linux tools (think PowerShell using where to filter the pipeline).
It is true that PowerShell gets a worse rap than it deserves for its names, though.
I misunderstood this project when I posted the original comment. This looks awesome! I think it’s time to look at shell scripting the same way, sh/bash syntax as awkward and python/ruby/etc. have to many dependencies and become unwieldy very quickly. This seems like a rabbit hole I could waste a lot of time on, so I should get started :-)
Ah, edged out. For the last two years I’ve been designing a shell with one of the primary features being the output of shell commands should be typed and tabular, to much the same effect as nushell achieves. I was going to write wrappers for the standard posix commands and a small suite of tools for tabularizing and working with the tabular data (along the lines of its lines, split-columns, etc.). In time, introduce a pair of stddata file descriptors to go alongside stdin and stdout. I guess this was an idea whose time had come. Good thing I still have more ideas, but I do wish I hadn’t had to work the last couple years so I could’ve been first.
I linked this at the bottom of my own proposal (which hasn’t been implemented yet)
PowerShell had this idea more than a decade ago, and it clearly isn’t the last word. It’s a hard problem and they didn’t get it right IMO (and apparently the authors of nushell agree).
About 10 years ago I read this 1991 paper about structured data in shell
That’s why it helps to work on your stuff in public. I don’t think it makes sense to be “scooped”, because these things take a long time to implement, and they have to interoperate with a lot of other software. Getting your software out there early means that’s more likely to happen. Release early, release often, etc.
Just because someone wrote a blog post about it something, doesn’t mean it works well and satisifies users. I know that from the other side with Oil :)
I’d be interested in seeing what you came up with. Here are my ideas:
I’ve also received e-mails from several people who are excited about putting structured data in Oil.
It seems to fall down on the details though. A lot of people have ideas in their heads, but it has to be prototyped to see if it even makes sense.
If you had a prototype, I’d be doubly interested in seeing it, and maybe incorporating those ideas into Oil! My very last blog post is explictly calling for other people to get involved in the design of the Oil language.
There’s some element of competition in open source, but there’s also collaboration! One thing I’ve definitely realized is that the job is so big that not one person can do it.
FWIW I plan to see if the nushell authors want to collaborate on shell-agnostic autocompletion, since that’s an area that all new shells need to tackle. Right now all upstreams target bash, and sometimes zsh. fish, zsh, elvish, etc. all have to “boil the ocean” on their own.
I later found that my wiki page already cited a paper (by the same authors) that was a precursor to the one that he was surprised to have been “edged out” by (so it seems to predate his by a bit).
So there is really no such thing as a grand new idea waiting to be implemented and unveiled to the world. It’s all built on top of other people’s work and ideas. Independent and multiple invention is a thing, but some things are also just “in the air”.
The “database” approach of Powershell makes it both me like and dislike it (same for nu). The problem is the meaning of the pipe.
I like a pipe to have the simplicity of just calling pipe(2), and I like data being passed in a consistent format between programs as well (think about how free(1) displays a different number of words according to the language of your operating systems!).
The easy and to me obvious solution:
Either keep it plain text: awk 'NR > 1 { print $2 }' and friends will play very nice with it. And fix all the commands that do not cope with it.
Rely on isatty(3) to either print some tsv format or some human-readable format. Maybe the shell can have an option to format all raw tsv output into tables.
The problem is that this means to edit each and every commands.
FreeBSD went toward somethibg like that with libxo (xo_emit and friends) to show various output format on all its coreutils (including err… you know XML ;_;, but also JSON). TSV would be fine really!
I love seeing projects inspired by PowerShell - a rich communication format was a great idea.
For my own use, I care less about ls, ps, and all the other trappings of a shell. I primarily want a data processing pipeline based around a common intermediary data format - that gets me over 90% of the benefit I saw in PowerShell, and still integrates easily with traditional POSIX shells. For anyone interested, I would recommend looking at RecordStream and miller - phenomenal tools based around a pipeline that works off of a rich, shared data format (eg. for recs, newline delimited json). Comparing the examples to what would be required in a standard, loosely structured text pipeline demonstrates the power of the idea.
This looks like a really cool project! I will definitely check it out.
With that said, I take issue with this from the end:
Nu would not have been possible without Rust. Internally, it uses async/await, async streams, and liberal use of “serde” to manage serializing and deserializing into the common data format and to communicate with plugins.
We also heavily leveraged crates.io. The ability to load numerous file formats, display messages, draw tables, and more all came from the hundreds (thousands?) of generous developers who wrote the crates we use in Nu.
Plenty of programming languages have async, serialization libraries and RPC, and package repositories full of nice libraries like that.
My thoughts exactly. The “would not have been possible” seems a bit exaggerated. “would not have been as fast/pleasant to develop/fun” might have been a better way to phrase it.
Also, regarding the second quoted statement, I’m not sure that my shell having so many dependencies on external packages is a good thing.
This looks very interesting! It’s a “query-like” approach to interactive shells, from what I gather. Not that it matters much, but is it POSIX compliant?
Oil is supposed to do most of what I see so far in nushell (probably minus the cd into a data structure), but I probably bit off more than I can chew. Help is appreciated :)
I did just start implementing the Oil language though, and it’s not too far from dictionaries / vectors to data frames.
Oil looks interesting. I would use any alternative shell that adds expressiveness as long as it were POSIX compliant. I never use bashisms except for the auto-complete scripts that come with my OS.
For me it’s not a matter of using bash OR powershell / bash OR nushell. It’s a matter of using “just bash” or “bash + powershell”. They augment bash rather than replace it.
It’s hard to switch because there’s a lot of other stuff that depends on bash.
there are thousands of lines completion logic for coreutils, etc.
Python’s virtualenv has a bunch of bash
nix-env is bash
asdf is bash
opam switch is bash (OCaml’s package manager)
I started the Oil shell with the idea of bash compatibility, but even I underestimated how much bash code there is out there!
Basically I want to use one shell and not two. Although honestly I can see why that might be too idealistic… plenty of people use two shells – zsh/fish interactively and bash for scripting / system shell seems to be a very common combo. But still I’m going to pursue replacing bash and then augmenting it :)
Well if it’s not my login shell (selected via chsh), then I guess it wouldn’t really matter. It would be the equivalent of dropping into a Python shell. To act as a login shell, it would need to be POSIX compatible because everything in a POSIX ecosystem expects that to be the case. It also would be tiresome to have to disregard all my previous knowledge to begin to even start using the shell.
If it’s not my login shell, I probably wouldn’t use it much and would opt for more general purpose languages like Python for the situations where I need a more expressive language.
This is cool, but doesn’t solve the biggest problem in CLI: escaping.
Figuring out how to escape something through a particular mode of Vim, then the shell, then whatever processes the command is so ugly. Maybe if I were Tim Pope, I’d know exactly where to put the slashes and quotes, but I’m not.
For everyone out there re-writing everything in Rust, let that problem sit in the back of your mind and chew on it for a while.
This rules. PowerShell had the right idea and executed on the Unix philosophy (small, composable programs which do one thing well) but hitched its horse to an overly-verbose C# API, awful names, awful error messages, and so on. It’s extremely good to see the same idea executed with what looks like a really nuanced understanding of where PowerShell comes up short in interacting with real data / external tools.
PowerShell has aliases to make it look similar to sh. For example, the famous word frequency problem in Powershell:
Having used and developed for PowerShell seriously for several years, “look similar to sh” is not enough, particularly when small incompatibilities and naming conflicts impede interop with common Linux tools (think PowerShell using
whereto filter the pipeline).It is true that PowerShell gets a worse rap than it deserves for its names, though.
I misunderstood this project when I posted the original comment. This looks awesome! I think it’s time to look at shell scripting the same way, sh/bash syntax as awkward and python/ruby/etc. have to many dependencies and become unwieldy very quickly. This seems like a rabbit hole I could waste a lot of time on, so I should get started :-)
Ah, edged out. For the last two years I’ve been designing a shell with one of the primary features being the output of shell commands should be typed and tabular, to much the same effect as nushell achieves. I was going to write wrappers for the standard posix commands and a small suite of tools for tabularizing and working with the tabular data (along the lines of its
lines,split-columns, etc.). In time, introduce a pair of stddata file descriptors to go alongside stdin and stdout. I guess this was an idea whose time had come. Good thing I still have more ideas, but I do wish I hadn’t had to work the last couple years so I could’ve been first.This is meant to be constructive: the “edged out” framing seems to indicate a misunderstanding of how open source works ?
I’d be interested in seeing what you came up with. Here are my ideas:
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
I’ve also received e-mails from several people who are excited about putting structured data in Oil.
It seems to fall down on the details though. A lot of people have ideas in their heads, but it has to be prototyped to see if it even makes sense.
If you had a prototype, I’d be doubly interested in seeing it, and maybe incorporating those ideas into Oil! My very last blog post is explictly calling for other people to get involved in the design of the Oil language.
There’s some element of competition in open source, but there’s also collaboration! One thing I’ve definitely realized is that the job is so big that not one person can do it.
FWIW I plan to see if the nushell authors want to collaborate on shell-agnostic autocompletion, since that’s an area that all new shells need to tackle. Right now all upstreams target bash, and sometimes zsh. fish, zsh, elvish, etc. all have to “boil the ocean” on their own.
https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal
FWIW here is a similar comment I wrote to somebody else who thought he had been “edged out” (my handle is chubot)
https://news.ycombinator.com/item?id=20265734 (original response)
https://news.ycombinator.com/item?id=20267099 (another one and my reply)
I later found that my wiki page already cited a paper (by the same authors) that was a precursor to the one that he was surprised to have been “edged out” by (so it seems to predate his by a bit).
So there is really no such thing as a grand new idea waiting to be implemented and unveiled to the world. It’s all built on top of other people’s work and ideas. Independent and multiple invention is a thing, but some things are also just “in the air”.
The “database” approach of Powershell makes it both me like and dislike it (same for nu). The problem is the meaning of the pipe.
I like a pipe to have the simplicity of just calling pipe(2), and I like data being passed in a consistent format between programs as well (think about how free(1) displays a different number of words according to the language of your operating systems!).
The easy and to me obvious solution:
awk 'NR > 1 { print $2 }'and friends will play very nice with it. And fix all the commands that do not cope with it.The problem is that this means to edit each and every commands.
FreeBSD went toward somethibg like that with libxo (xo_emit and friends) to show various output format on all its coreutils (including err… you know XML ;_;, but also JSON). TSV would be fine really!
I love seeing projects inspired by PowerShell - a rich communication format was a great idea.
For my own use, I care less about ls, ps, and all the other trappings of a shell. I primarily want a data processing pipeline based around a common intermediary data format - that gets me over 90% of the benefit I saw in PowerShell, and still integrates easily with traditional POSIX shells. For anyone interested, I would recommend looking at RecordStream and miller - phenomenal tools based around a pipeline that works off of a rich, shared data format (eg. for recs, newline delimited json). Comparing the examples to what would be required in a standard, loosely structured text pipeline demonstrates the power of the idea.
https://github.com/benbernard/RecordStream
https://github.com/johnkerl/miller
This looks like a really cool project! I will definitely check it out.
With that said, I take issue with this from the end:
Plenty of programming languages have async, serialization libraries and RPC, and package repositories full of nice libraries like that.
My thoughts exactly. The “would not have been possible” seems a bit exaggerated. “would not have been as fast/pleasant to develop/fun” might have been a better way to phrase it.
Also, regarding the second quoted statement, I’m not sure that my shell having so many dependencies on external packages is a good thing.
This looks very interesting! It’s a “query-like” approach to interactive shells, from what I gather. Not that it matters much, but is it POSIX compliant?
Regarding the use of the word nu in nushell - I’ll just leave this here: https://github.com/programming-nu/nu
Interesting concept but pragmatically, I can’t use it if it’s not POSIX compliant
FWIW Oil is POSIX compliant (and it doesn’t feel too far from exceeding the POSIX compliance of bash with recent help by the Smoosh project )
I have designs for structured data, but they aren’t implemented.
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
Oil is supposed to do most of what I see so far in nushell (probably minus the
cdinto a data structure), but I probably bit off more than I can chew. Help is appreciated :)I did just start implementing the Oil language though, and it’s not too far from dictionaries / vectors to data frames.
Oil looks interesting. I would use any alternative shell that adds expressiveness as long as it were POSIX compliant. I never use bashisms except for the auto-complete scripts that come with my OS.
I’m genuinely curious, why?
For me it’s not a matter of using bash OR powershell / bash OR nushell. It’s a matter of using “just bash” or “bash + powershell”. They augment bash rather than replace it.
It’s hard to switch because there’s a lot of other stuff that depends on bash.
For example
I started the Oil shell with the idea of bash compatibility, but even I underestimated how much bash code there is out there!
Basically I want to use one shell and not two. Although honestly I can see why that might be too idealistic… plenty of people use two shells – zsh/fish interactively and bash for scripting / system shell seems to be a very common combo. But still I’m going to pursue replacing bash and then augmenting it :)
Well if it’s not my login shell (selected via
chsh), then I guess it wouldn’t really matter. It would be the equivalent of dropping into a Python shell. To act as a login shell, it would need to be POSIX compatible because everything in a POSIX ecosystem expects that to be the case. It also would be tiresome to have to disregard all my previous knowledge to begin to even start using the shell.If it’s not my login shell, I probably wouldn’t use it much and would opt for more general purpose languages like Python for the situations where I need a more expressive language.
You can’t use Software if it’s POSIX compliant?
If it’s not POSIX compliant.
Oops, you’re right.
This is cool, but doesn’t solve the biggest problem in CLI: escaping.
Figuring out how to escape something through a particular mode of Vim, then the shell, then whatever processes the command is so ugly. Maybe if I were Tim Pope, I’d know exactly where to put the slashes and quotes, but I’m not.
For everyone out there re-writing everything in Rust, let that problem sit in the back of your mind and chew on it for a while.