The article mentions that Coreutils used to be 3 separate bundles, and I fell down a little bunnyhole. Here is the composition of Fileutils (2001), Textutils (2003), and Shellutils (2000), with thanks to the Tucows collection as preserved by the Internet Archive:
I wish Rust had a bigger standard library (“batteries” included, like python in some degree)
See for example sort. I realise all of us download and run programs with lots of dependencies most days but I feel like core utils should not pull non-standard dependencies.
Note that among those 12 direct dependencies, Python’s stdlib has direct equivalents only to 4: clap, itertools, rand, tempfile. Things like unicode-width, rayon, semver, binary-heap-plus are not provided by Python. compare, fnv, memchr and ouroboros are somewhat hard to qualify Rust-isms.
In addition, it’s worth noting that a lot of projects eschew argparse (what the alternative to clap would be) for click. If a similar project was done in python, I’d almost bet money that they’d use click.
rand being separate has some advantages, largely that it is able to move at a pace that’s not tied to language releases. I look at this as a similar situation that golang’s syscall package has (had? the current situation is unclear to me rn). If an OS introduces a new random primitive (getrandom(2), getentropy(2)), a separate package is a lot easier to update than the stdlib, which is tied to language releases.
Golang’s syscall package has (had?) a similar problem, which led to big changes being locked down, and the recommended pkg being golang.org/x/sys. There’s a lot more agility to be had to leverage features of the underlying OS if you don’t tie certain core features to the same cadence as a language release. (this is not to say that this is the only problem with the syscall package being in the stdlib, but it’s definitely one of them. more info on the move here: https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit)
I’d use getopt over argparse. argparse just has really abysmal parsing which is different from other shell tools, especially when dealing with subcommands.
This problem is Software Engineering Complete (like NP-Complete - transformable to any other SW Eng Complete thing). As just one other example, Nim also struggles with what should be in the stdlib vs. external packages. Rust has just about 1000x the resources than Nim for a much more spartan core stdlib, but of course the Nim stdlib almost surely has more bugs than that Rust spartan core. So, a lot of this boils down to A) tolerance for bugs, B) resources to maintain going forward, and C) the cost of complexity/generality in the first place, and probably a factor or two I’m forgetting/neglecting. ABC relate far more to community & project management/attitudes than language details themselves. Also, presence in the stdlib is not a panacea for discoverability because as the stdlib grows more and more giant, discoverability crashes.
Note this is neither attack nor defense but elaboration on why this is not an easy problem.
I wonder if the reason nim has to have a big standard library is to attract people. Rust already has the following, as you said, and people are sure to create an kinds of things. Whereas, if one was to try nim, if there wasn’t the stdlib, they would have to do everything on their own.
Same. A big part of the learning curve for me was discovering modules like serde, tokio, anyhow/thiserror, and so on that seem necessary in just about every Rust program I write.
Yeah i’ve used it and it felt refreshing :) But for other small tools perhaps having a reference and minimal implementation would be good. I like the smol crate and I think would be perfect for this
All of them developed over time and became a de-facto standard. But it was always the intention that the std doesn’t try to develop these tools as you need some iterations, which won’t work on with a stability guarantee. tokio just went to 1.0 this? year, I’ve got code lying around using 0.1,0.2 and some 0.3 (and don’t forget futures etc).
anyhow/thiserror ? Well there is failure,error-chain,quick-error,snafu,eyre (stable-eyre,color-eyre), simple-error….. And yes, some of them are still active as they solve different problems (I specifically had to move away from thiserror) and some are long deprecated. So there was a big amount of iteration (and some changes to the std Error trait as a result).
You don’t want to end up like c++ (video) where everybody treats the std implementation of regex as something you don’t ever want to use.
There is problem with such approach of “batteries included” in the standard library - development of such libraries slows down or stagnates. Actually I prefer to use set of well behaved external libraries than need to replace “built-ins” because these are too simplified for any reasonable usage.
The thing is that as a user I “trust” standard rust-lang crates at first glance; surely if I check the external libraries out or recognize them I will know they are well behaved and performant. Trust and trusting trust is such a big problem in software in general.
And point me to one, just one, example where it worked? If something is merged into the core, then it will die there. Python has examples of such code, Ruby has examples of such code, etc. How often for example built in HTTP client is good enough to be used in any serious case? How often instead you pull dependency to handle the timeouts, headers, additional HTTP versions, etc. better/faster/easier?
The rust ecosystem, IMO, is far too eager to pull in third party dependencies. I haven’t looked deep into this tool, but a quick glance leads me to believe that many of these dependencies could be replaced with the standard library and/or slimmed down alternative libraries and a little extra effort.
Unfortunately it’s not always that simple. Let’s see the third party dependencies I pulled for meli, an email client, which was a project I started with the intention of implementing as much as possible myself, for fun.
From a quick glance, only nix, linkify, notify, uuid, bitflags could be easily replaced by invented here code because the part of the crates I use is small.
Definitely agree that it isn’t that simple, and each project is different (and often it’s not worth the energy, esp for applications, not libraries), but it’s something I notice in the Rust ecosystem in general.
But then you’re getting less popular deps, with fewer eyeballs on them, from less-known authors.
Using bare-bones pico-args is a poor deal here — for these CLI tools the args are their primary user interface. The fancy polished features of clap make a difference.
Why do you think an external merge sort should be part of the Rust stdlib? I don’t think it’s part of the Python stdlib either. Rust already has sort() and unstable_sort() in its stdlib (unstable sort should have been the default, but that ship has sailed).
Does anyone here have experience with this on Windows? I’ve been having a lot of trouble getting Bash for Windows or WSL working on one of three laptops (the other two it worked fine). This might be a good alternative since I’m just trying to have some basic build scripts (cp, mv, etc.) that work on Linux and Windows.
TBH, I’d use Python for simple/basic scripts for cross platform build/install stuff, if you’re not going to do dual bat/bash files, as a lot of work has gone into making Python work on windows a lot better.
I did end up writing a simple lexer in Python and switch on the command passed for stuff that doesn’t exist exactly the same in Powershell like rm -rf.
What program is that? I don’t have Windows, but I believe installing git will give you access to bash (IIRC, this option is checked by default during installation, but you could make sure it is selected if you try this option).
I am confused by MS docs so I cannot tell if Bash for Windows is WSL or if it’s something else.
I did try installing git but bash only works in its own window that is not a real terminal emacs tells me. I cannot run bash in Powershell through the git install.
“Bash for Windows” == Ubuntu == WSL. WSL1 is like reverse WINE while WSL2 is a full-blown VM. Both of them need some OS/machine support that your computer might not have enabled.
git bash == MSYS2 bash == Cygwin bash. It’s a bog standard program that links with cygwin1.dll. You can run bash from cmd.exe or powershell.exe just by exposing bash in your path. But when you use the desktop shortcut to start git bash, that actually runs bash in the Mintty terminal (comes with your git install and emulates Unix-isms), which Emacs says is “not a real terminal” (because it’s not the Windows one that Emacs is expecting).
But you can run Emacs on a “real terminal” too. cmd.exe and powershell.exe (and bash, if you run it that way) are shells that run on conhost.exe (comes from Windows), which is “a real terminal”. I haven’t tried it in years, but I think Vim works as is on conhost.exe but Emacs might need/want a different $TERM variable. In either case, Emacs has a complicated enough TUI that it doesn’t work well with the limited capabilites of conhost.exe. (MS has come up with a much nicer Windows Terminal in recent years, but it’s not quite ready to replace conhost.exe as the default, and it’ll have to deal with backward compatibility forever and ever as all things MS.)
If you just want to have some basic bash/make stuff that work on Linux and Windows, use MSYS2 (which is what your git install does). You can install MSYS2, but git for Windows also bundles a copy of MSYS2. By default, git for Windows keeps everything out of your path, but you can just put it in your path as needed (the installer has an option to do that, but warns you that name clashes/shadowing will occur with Windows programs with the same names, which can come up if you run .bat files that use those programs).
Unless you’re using a POSIX for Windows distribution like Cygwin, it is almost certainly WSL-based, which means it’s virtualized. That’s why you can’t just run it like any other executable.
The article mentions that Coreutils used to be 3 separate bundles, and I fell down a little bunnyhole. Here is the composition of Fileutils (2001), Textutils (2003), and Shellutils (2000), with thanks to the Tucows collection as preserved by the Internet Archive:
Textutils: cat, cksum, comm, csplit, cut, expand, fmt, fold, head, join, md5sum, nl, od, paste, pr, sort, split, sum, tac, tail, tr, unexpand, uniq, wc.
Fileutils: ansi2knr, chgrp, chown-core, chmod, chown, cp, copy, cp-hash, dd, df, ls, ls-dir, dircolors, du, install, ln, ls-ls, mkdir, mkfifo, mknod, mv, remove, rm, rmdir, shred, sync, touch, ls-vdir
Shutils: basename, chroot, date, dirname, echo, env, expr, factor, false, groups, hostname, id, logname, nice, nohup, pathchk, printenv, printf, pwd, seq, sleep, stty, su, tee, test, true, tty, uname, users, who, whoami, yes.
Wouldn’t it make more sense to rewrite systemd in rust? It’s a bigger potential attack surface.
Seems like the work in vaguely that direction is underway: https://github.com/systemd/systemd/pull/19598.
I wish Rust had a bigger standard library (“batteries” included, like python in some degree)
See for example sort. I realise all of us download and run programs with lots of dependencies most days but I feel like core utils should not pull non-standard dependencies.
Note that among those 12 direct dependencies, Python’s stdlib has direct equivalents only to 4: clap, itertools, rand, tempfile. Things like unicode-width, rayon, semver, binary-heap-plus are not provided by Python. compare, fnv, memchr and ouroboros are somewhat hard to qualify Rust-isms.
In addition, it’s worth noting that a lot of projects eschew
argparse(what the alternative to clap would be) forclick. If a similar project was done in python, I’d almost bet money that they’d useclick.randbeing separate has some advantages, largely that it is able to move at a pace that’s not tied to language releases. I look at this as a similar situation that golang’ssyscallpackage has (had? the current situation is unclear to me rn). If an OS introduces a new random primitive (getrandom(2),getentropy(2)), a separate package is a lot easier to update than the stdlib, which is tied to language releases.Golang’s
syscallpackage has (had?) a similar problem, which led to big changes being locked down, and the recommended pkg beinggolang.org/x/sys. There’s a lot more agility to be had to leverage features of the underlying OS if you don’t tie certain core features to the same cadence as a language release. (this is not to say that this is the only problem with thesyscallpackage being in the stdlib, but it’s definitely one of them. more info on the move here: https://docs.google.com/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs/edit)I’d use
getoptoverargparse.argparsejust has really abysmal parsing which is different from other shell tools, especially when dealing with subcommands.True. Could be just rust-lang crates like futures-rs or cargo instead of being in the stdlib.
This problem is Software Engineering Complete (like NP-Complete - transformable to any other SW Eng Complete thing). As just one other example, Nim also struggles with what should be in the stdlib vs. external packages. Rust has just about 1000x the resources than Nim for a much more spartan core stdlib, but of course the Nim stdlib almost surely has more bugs than that Rust spartan core. So, a lot of this boils down to A) tolerance for bugs, B) resources to maintain going forward, and C) the cost of complexity/generality in the first place, and probably a factor or two I’m forgetting/neglecting. ABC relate far more to community & project management/attitudes than language details themselves. Also, presence in the stdlib is not a panacea for discoverability because as the stdlib grows more and more giant, discoverability crashes.
Note this is neither attack nor defense but elaboration on why this is not an easy problem.
I wonder if the reason nim has to have a big standard library is to attract people. Rust already has the following, as you said, and people are sure to create an kinds of things. Whereas, if one was to try nim, if there wasn’t the stdlib, they would have to do everything on their own.
Same. A big part of the learning curve for me was discovering modules like serde, tokio, anyhow/thiserror, and so on that seem necessary in just about every Rust program I write.
Not providing a standard executor was the only complaint I had from async Rust.
I like that there is no built-in blessed executor - it keeps Rust runtime-free.
I’ve worked on projects where using an in-house executor was a necessity.
Also gtk-rs supports using GTK’s event loop as the executor and it’s very cool to await button clicks :)
Yeah i’ve used it and it felt refreshing :) But for other small tools perhaps having a reference and minimal implementation would be good. I like the
smolcrate and I think would be perfect for thisAll of them developed over time and became a de-facto standard. But it was always the intention that the std doesn’t try to develop these tools as you need some iterations, which won’t work on with a stability guarantee. tokio just went to 1.0 this? year, I’ve got code lying around using 0.1,0.2 and some 0.3 (and don’t forget futures etc).
anyhow/thiserror ? Well there is failure,error-chain,quick-error,snafu,eyre (stable-eyre,color-eyre), simple-error….. And yes, some of them are still active as they solve different problems (I specifically had to move away from thiserror) and some are long deprecated. So there was a big amount of iteration (and some changes to the std Error trait as a result).
You don’t want to end up like c++ (video) where everybody treats the std implementation of regex as something you don’t ever want to use.
There is problem with such approach of “batteries included” in the standard library - development of such libraries slows down or stagnates. Actually I prefer to use set of well behaved external libraries than need to replace “built-ins” because these are too simplified for any reasonable usage.
The thing is that as a user I “trust” standard rust-lang crates at first glance; surely if I check the external libraries out or recognize them I will know they are well behaved and performant. Trust and trusting trust is such a big problem in software in general.
Yes, that why there are some “blessed” crates as well as there is Crev project to expand the trust.
just version the standard library interface, though
And point me to one, just one, example where it worked? If something is merged into the core, then it will die there. Python has examples of such code, Ruby has examples of such code, etc. How often for example built in HTTP client is good enough to be used in any serious case? How often instead you pull dependency to handle the timeouts, headers, additional HTTP versions, etc. better/faster/easier?
The rust ecosystem, IMO, is far too eager to pull in third party dependencies. I haven’t looked deep into this tool, but a quick glance leads me to believe that many of these dependencies could be replaced with the standard library and/or slimmed down alternative libraries and a little extra effort.
Unfortunately it’s not always that simple. Let’s see the third party dependencies I pulled for meli, an email client, which was a project I started with the intention of implementing as much as possible myself, for fun.
From a quick glance, only
nix,linkify,notify,uuid,bitflagscould be easily replaced by invented here code because the part of the crates I use is small.I cannot reasonably rewrite:
You could reduce transitory dependencies with:
serde -> nanoserde
structopt -> pico-args
Definitely agree that it isn’t that simple, and each project is different (and often it’s not worth the energy, esp for applications, not libraries), but it’s something I notice in the Rust ecosystem in general.
But then you’re getting less popular deps, with fewer eyeballs on them, from less-known authors.
Using bare-bones pico-args is a poor deal here — for these CLI tools the args are their primary user interface. The fancy polished features of clap make a difference.
Why do you think an external merge sort should be part of the Rust stdlib? I don’t think it’s part of the Python stdlib either. Rust already has sort() and unstable_sort() in its stdlib (unstable sort should have been the default, but that ship has sailed).
It’s funny how they “stole” GPL licensed tests, to test their MIT code.
How did they “steal” them?
Corporate espionage. Kidding, they just copied the source.
Got a link to where they copied it? Their tests directory is not filled with source from GNU tests.
I can’t wait to say “acktually it’s Rust/Linux”.
“Acktually,” I don’t think it works that way, or we’d call it C/Linux, right? :)
This sent me on a dive into features that the Coreutils maintainers have rejected, and:
This seems like a… weird rationale.
Does anyone here have experience with this on Windows? I’ve been having a lot of trouble getting Bash for Windows or WSL working on one of three laptops (the other two it worked fine). This might be a good alternative since I’m just trying to have some basic build scripts (cp, mv, etc.) that work on Linux and Windows.
It looks like Bash For Windows is WSL based.
TBH, I’d use Python for simple/basic scripts for cross platform build/install stuff, if you’re not going to do dual bat/bash files, as a lot of work has gone into making Python work on windows a lot better.
I did end up writing a simple lexer in Python and switch on the command passed for stuff that doesn’t exist exactly the same in Powershell like rm -rf.
What program is that? I don’t have Windows, but I believe installing git will give you access to
bash(IIRC, this option is checked by default during installation, but you could make sure it is selected if you try this option).I am confused by MS docs so I cannot tell if Bash for Windows is WSL or if it’s something else.
I did try installing git but bash only works in its own window that is not a real terminal emacs tells me. I cannot run bash in Powershell through the git install.
“Bash for Windows” == Ubuntu == WSL. WSL1 is like reverse WINE while WSL2 is a full-blown VM. Both of them need some OS/machine support that your computer might not have enabled.
git bash == MSYS2 bash == Cygwin bash. It’s a bog standard program that links with cygwin1.dll. You can run bash from cmd.exe or powershell.exe just by exposing bash in your path. But when you use the desktop shortcut to start git bash, that actually runs bash in the Mintty terminal (comes with your git install and emulates Unix-isms), which Emacs says is “not a real terminal” (because it’s not the Windows one that Emacs is expecting).
But you can run Emacs on a “real terminal” too. cmd.exe and powershell.exe (and bash, if you run it that way) are shells that run on conhost.exe (comes from Windows), which is “a real terminal”. I haven’t tried it in years, but I think Vim works as is on conhost.exe but Emacs might need/want a different $TERM variable. In either case, Emacs has a complicated enough TUI that it doesn’t work well with the limited capabilites of conhost.exe. (MS has come up with a much nicer Windows Terminal in recent years, but it’s not quite ready to replace conhost.exe as the default, and it’ll have to deal with backward compatibility forever and ever as all things MS.)
If you just want to have some basic bash/make stuff that work on Linux and Windows, use MSYS2 (which is what your git install does). You can install MSYS2, but git for Windows also bundles a copy of MSYS2. By default, git for Windows keeps everything out of your path, but you can just put it in your path as needed (the installer has an option to do that, but warns you that name clashes/shadowing will occur with Windows programs with the same names, which can come up if you run .bat files that use those programs).
Unless you’re using a POSIX for Windows distribution like Cygwin, it is almost certainly WSL-based, which means it’s virtualized. That’s why you can’t just run it like any other executable.