I had been imagining something like the ability to extend switch to operate on any aggregate type that has a hash-function definition available, but having a comparator generalizes that, and even supports some interesting use-cases like canonicalization-in-comparison (as @andrewrk asked about) as well as even fancier things like edit distance matching!
NB: I specifically don’t want to get into any political discussion, merely a statistical one.
That said, surely with 50% of the respondents self-reporting as belonging to the LGBTQIA+ communit{y,ies} this indicates a statistically significant selection bias in the targeted users? I’m not complaining about (in)equity but I am questioning whether this fairly “purely demographic” question ends up indicating that the answers for the remaining questions may not hold for “the average rust user” as rust has clearly escaped niche status and therefore one would assume that demographic distribution of users skews somewhat closer to the baseline distribution of (system or backend) developers in general, where you’d expect that number to be smaller (though by how much apparentlyvaries).
Lest one consider this is still an attempt to disguise rage politics, I want to give a specific point: I think the avenues where this survey was being suggested might skew to a specific demographic (not in terms of gender identity specifically, but also wrt nature of $work, age, experience, etc) and exclude a good chunk of potentially relevant (to rust maintainers) users. With a sample size of one: I saw (and believe I participated in?) the 2023 survey but I didn’t hear about the 2024 survey despite being on the rust github and associated blogs “all the time”, but one likely explanation is that I quit reddit (and therefore r/rust) in 2023. And as another point, I know the most tech-active mastodon nodes just happen to be most active on the lgbtq+ mastodon servers.
Not all questions were answered by all participants of the survey. Note that there were 1110 total responses to that question. The interpretation should be that 50% of the people who consider themselves to be a part of an underrepresented and/or marginalized community report as being LGBTQIA+.
I think the blog post does a very good job at explaining that:
We also asked whether respondents consider themselves members of a marginalized community. Out of those who answered, 74.5% selected no, 15.5% selected yes, and 10% preferred not to say.
We have asked the group that selected “yes” which specific groups they identified as being a member of. The majority of those who consider themselves a member of an underrepresented or marginalized group in technology identify as lesbian, gay, bisexual, or otherwise non-heterosexual. The second most selected option was neurodivergent at 46% followed by trans at 35%.
I thought the survey was well-publicized — I found out about it by following the official Rust blog in my RSS reader. I acknowledge you may not be trying to disguise rage politics, but you do seem to be implying very strongly that there was some sort of an attempt to bias the survey towards LGBTQIA+ participants without understanding the basic results of the survey.
Even with that adjustment, one point that I find very interesting is that 15.5% times 35.8% (people who answered “trans”) works out to an overall fraction of 5.5%. That’s 10x higher than the general population.
As a trans person who knows a lot of trans people who use Rust, this doesn’t entirely surprise me, but it’s slightly mind-blowing and also probably at least somewhat affected by selection bias.
Thanks for pointing out that possible statistical explanation for the discrepancy I noted; it potentially answers my question in full.
(I don’t see how any part of my post could be interpreted as accusing the rust org to try to intentionally skew towards certain users; for the record I hold the maintainers in very high regard. I thought it was clear I was only reaching for possible factors that could explain the demographics of the respondents. If I said something along the lines of “they clearly should have used xxx as a venue”, then maybe; I only indicated my activity on certain hubs as a signal of how keyed in to the language I personally was, for the benefit of readers of my comment. And my comment certainly should not be read to imply that I have any problems with any demographic group, because it certainly doesn’t say that and I certainly don’t.)
I wasn’t saying that you were accusing the Rust maintainers of skewing the data, just that your comment sounded like it was hinting at skewing the data towards certain demographic without fully appreciating the results.
That said, I have no reason to doubt your goodwill, I share your high regard for the maintainers, and I’m glad there was a satisfactory explanation for your question!
To just put the numbers together, 7310 completed surveys, 1110 * 54.1% users reporting as LGBTQ, is 8% of respondents choosing to report as LGBTQ. Combined with the 10% who preferred not to say that’s 8 to 18% of respondents are LGBTQ, I’d guess somewhere towards the lower end of that range.
Lovely read and amazing perseverance and persistence… but the conclusion/benchmark missed the biggest wins. No mainstream linker supports incremental linking (not even mold) and that is where the biggest gains lie. With a real world codebase, for example, linking fish-shell takes 1-3 min per binary (there are three of them) in release mode; this applies even if you change a single character in the core codebase. While normally the edit-debug loop is done in debug mode, when you are editing for performance you want release builds (and we do that often). Prior to switching to rust (and static linking), our C++ codebase would (dynamically) link the binaries in a couple of seconds at most.
The secret is that one says third party projects need to live in your monorepo. If you use a sane language with support for pinning, the only time you have third party source code you need to include in your repo is when you’ve forked it - i.e. when it’s no longer third party by definition.
Things like hosting your own module/package/nuget/cargo/whatever cache servers in-house (optionally restricted only to package/version combos you’ve vetted for license and security compliance) takes you the rest of the way there.
This is just moving from the unix-style cli to the old win32-style cli. Except Microsoft gave up and ended up embracing ANSI escapes and CSIs (some time after WSL landed) to support the ever-growing corpus of apps specifically targeting terminal emulators.
In the Win32 console model, apps emitted only text to stdout and used the console handle to do everything else from setting attributes to switching between the full screen buffers. And the open source world made fun of them for it.
And the open source world made fun of them for it.
I wouldn’t do that, it isn’t like the unix terminal design is exactly ideal either.
I don’t think the unix design is as offensive on the output side as the input though; if you’re just writing a program that wants to output some colorful text, it isn’t all bad. Just remember to check isatty and the command line flag that overrides that before adding the color sequences!
On the input side though, yikes, what a pile of disgusting hacks. You have to do a timeout check on the esc char! And woe be upon you if you need to query the terminal for something, since now that’s both input and output sequences! Worth noting a lot of the newer extended terminal emulators are introducing new features to change this stuff, but they’re tied by the need to be compatible with the large existing ecosystem of applications.
There’s also the separate question of api vs underling data too. In the underlying data, one stream does make synchronization and redirection and other things easier. But then in the api you might want it separate anyway, to make supporting --color=auto and --color=off easier. Stripping escape sequences from the output isn’t awful though, so you could just always write them in your code and have the library function just does that.
Nevertheless, I’m not a fan of making fun of designs… there’s often fair reasons for differences that you can learn from.
If you generalise to that level, well, duh. The argument that in-band signalling UI is harmful in the same way smashing strings together versus prepared statements is not a difficult one.
Microsoft didn’t “give up” – it was a no brainer for them to throw a few idle headcounts on writing an emulator for their WSL trajectory and getting more developers into VScode. They’re otherwise busy adding AI keys and ads in the UI so perhaps their sensibilities is not something to use as ‘appeal to authority’ or inspiration.
I am intimately familiar with the inner workings of the different console layers in the MS ecosystem and their reasons for being – from the days of conio.h, int 21h, mode con codepage prepare and onwards. Their powershell model is much closer in comparison but the power they had there came specifically from them having something of a functional IPC system to work with. Linux/BSD isn’t so lucky.
Maybe it’s a “this old 10/100 network hardware bugs out in this configuration unless you use a 10Mbit switch or a managed switch with the port forced to 10Mbit operation” situation.
Ah, but don’t you know, they “aren’t” rolling their own crypto, per their FAQ.
Is Session rolling its own cryptography?
No, Session does not roll its own cryptography. Session uses Libsodium, a highly tested, widely used, and highly-regarded crypto library.
Libsodium is completely open-source.
heavily rolls eyes
I like libsodium. It’s a great library of cryptography algorithms.
It doesn’t come, batteries included, with a full protocol for end-to-end encryption built-in. And so anyone who uses libsodium for e2ee is necessarily rolling their own crypto.
I’ve only heard of Session in passing over the years. This result is not surprising.
i initially thought you were being overzealous, until i read the first paragraph of the article
using libsodium to implement the documented signal protocol, i think would be fine. it does have risks, and should have some review before relying on it, but i wouldn’t really call that “rolling your own crypto”. and having a clean-room re-implementation would probably be good for the ecosystem
…but that’s not what they’re doing. they’re implementing their own protocol, and a cursory glance at their reasoning suggests that they want a decentralized messenger and care about security as an afterthought. which would be fine if they branded it that way, and not as an alternative to signal
This may be a little off topic, but I dislike the phrase “don’t roll your own crypto”.
Don’t roll your own crypto is generally a term which in itself is very ambiguous.
I’ve seen this phrase being thrown around when people just use gnutls in C vs people implementing some hash algorithm themselves. One I find very valid while the other one is an area where I would just use libsodium.
There are so many layers in crypto where you can apply the phrase that I find refuting (their claims with) this phrase in itself is meaningless unless you know what the authors intended. In this case it may as well be claims regarding the resistance against certain side channel attacks.
I’ve always asked myself how I can identify the moment I arrive at a skill level where I’m allowed to “roll my own crypto” depending on each possible interpretation people are using.
There are so many layers in crypto where you can apply the phrase that I find refuting this phrase in itself is meaningless unless you know what the authors intended.
Absolutely. And the advice, taken to its logical extreme, would result in zero cryptography ever being developed.
It’s supposed to be along the same lines as advice given from lawyers to their kids that are considering a career in law. They say, “Don’t be a lawyer,” and if their kid isn’t dissuaded and can argue why they’d succeed as a lawyer, then maybe they should be one.
“Don’t roll your own crypto” is along the same lines. I currently do it professionally, but I also have a lot of oversight into the work I do to ensure it’s correct. Detailed specifications, machine-assisted proofs, peer review, you name it. I never start with code; I always start with “here’s why I’m doing this at all” (which includes closely related ideas and why they do not work for the problem I’m solving) and a threat model for my specific solution.
It can take months, or even years, to get a new cryptography design vetted and released with the appropriate amount of safety.
When it comes to cryptography protocol design, the greatest adversary is often your own ego.
I always read the advice as an incomplete sentence, which ends with “unless you know what you’re doing”, which is coincidentally like other safety advice, right? “This thing that you’re about to do is risky and dangerous unless you know how to do it, and in some cases, even if you do. Avoid doing it if you can. Exercise caution and care otherwise.” No?
I always viewed it as “don’t ship your own” - feel free to roll your own to play around with, but be cautious and get some review before putting it into production.
One piece of advice I’ve heard is: Before trying to build crypto, learn how to break crypto. Cryptopals is a good resource for that. It’s mindbending to learn about all the weird ways that crypto can fall apart.
I remember many moons ago that an expert in security and crypto actually published a list of cryptographic choices that should be your default. I wonder if this rings a bell to someone, it would be nice to recover that document, publish it here and see what this community would say in terms of freshen it up.
I might be wrong, but I think in the beginning the meaning of the phrase “don’t roll your own crypto” mean “do not try to come up with cryptographic algorithms on your own; use something tested and done by someone who know what they are doing”. I think the best way to describe what Soatok is putting forward is “don’t skip the default practices of security” or “don’t wave away cryptographic defaults in name of a watered down threat model”.
I think that’s a very reasonable concern. Particularly in light of the very first issue @soatok cites: the removal of PFS from the protocol. I’m on record as being skeptical of the “just use signal” advice that seems frequently deployed as a though-terminating measure in discussions about encrypted communication, but if I wanted to make something that was like signal but really a deniable honeypot, Session makes the same choices I would. It seems like a downgrade from signal in every important way.
Unrelated: the “Ad-blocker not detected” message at the bottom of the post made me laugh quite a bit. I use some tracker-blocking extensions (and browser configs) but I don’t run a real ad blocker in this browser. But many sites give me an “ad blocker detected” message and claim I need to turn off my non-existent ad blocker to support them. This site telling me I’m not running enough of one struck me as very funny.
Sure, its plausible.
But I find basically every time Soatok (or any security researcher) exposes any application that advertises itself as “secure/private” on the box, for their glaring bad practices, people (myself included) immediately go to “this is so stupid it has to be a honeypot”.
Are they all honeypots? (genuinely, maybe yes), or is it just stupidity?
I would posit stupidity. Previous honeypots that weren’t takeovers of server operators have been somewhat targeted: Anom required a buy-in of the phone (as evidence you’re a criminal), Playpen required you be a pedophile (or at least, hanging out online with pedophiles) to be caught in the net, Hansa was a drug market, etc. Creating a general-purpose backdoored app to en masse catch criminals seems to cast quite a wide net when the same arrest portfolio can probably be gathered by doing the same thing to Twitter DMs with a backdoor and a secret court order. I wouldn’t put it past law enforcement but it seems like a mega hassle vs. targeted honeypots and backdoors.
If it were a honeypot (or backdoor), it’s certainly too much hassle for legitimate law enforcement purposes like the ones you described. You’d want this for someone you couldn’t reach through normal court (even a secret rubberstamp like FISA) channels.
This would be more like something you’d use for getting information from a group operating under some legal regime that’s not friendly to you gathering that information. Getting it in place, then convincing the group you were interested in to migrate off, say, Telegram, might be one approach.
The interesting thing in this case (IMO) here is that the fork removes things that were:
Already implemented by an easy-to-reuse library
Not costly in terms of performance or cognitive overhead
Seemingly beneficial to the stated goals of their product
and without articulating the upside to their removal very persuasively. Sure, stupidity is always a possibility. But it feels more like they want to add some features that they don’t want to talk about. On the less-nefarious end of that spectrum, I could imagine that it is as simple as supporting devices that don’t work with the upstream, but that they don’t want to discuss in public. It’s also easy to imagine wanting to support some middle scanner-type box on a corporate network that PFS would break. But it could also be something like being able to read messages from a device where you can maintain passive traffic capture/analysis but can’t (or would prefer not to) maintain an ongoing endpoint compromise without detection. e.g. You have compromised a foreign telco and can pull long term key material off a device when its owner stays in your hotel, but you can’t or won’t leave anything running on there because the risk of detection would then be too high.
That’s just all speculation about when it might serve someone with an effectively unlimited budget to do something like this. Stupidity is certainly more common than such scenarios.
Only the first bit could charitably be attributed to “don’t roll your own crypto”. The rest was just obtuse idiocy or malevolence. Call the library-provided “encrypt this chunk with symmetric encryption using this key” then providing a public key.. that’s not about rolling your own crypto.
Really cool and it’s nice how easy this is to pull off under Linux in user space. I can’t begin to think how many drivers I believe I’d need to write to do this on Windows.
One nitpick with regards to the invocation of Shannon’s law: the 1.5Mbps isn’t the max theoretical channel limit; that’s the max digital rate limit of the underlying digital tunnel OP chose (raw pcm, 16-bit, 48MHz, etc).
You could certainly do much better performance-wise and blow past the so-called 100% limit named in the post with a modern modulation scheme instead of the basic differential Manchester/bi-phase mark coding used by audio standards to define the modulation at the physical layer over toslink fiber. I think there are plenty of quadrature amplitude modulation open source libraries you could stick in there as a start, but you’d have to read/write raw signals to the toslink channel, not going through arecord and aplay.
I’ve used fish as my main shell for something like 13 years and while I have no real opinons of a rust rewrite. I’ve been pleasently surprised by the stability of the shell, since its been a small hobby project of someone I dont know, I’ve granted it a crash and bug allowance. However, fish has never crashed on me nor have I experienced any bugs that made me drop it even for a day.
If rewriting in rust is what the team thinks is right I trust you and wish you keep your fun in making it. I’ve loved using it and I hope you don’t lose interest any time soon.
Thanks for your support and patronage! We don’t generally release beta builds (our philosophy is to keep master as stable as the last release or better) but this has been an exceptional process and we’re hoping that with a beta release we can catch any real-world crashes we know must have been introduced but not yet caught, thanks to the laws of statistical probability. So if fish does crash for you, please do let us know and rest assured that we are doing our best to make that as unlikely as possible.
Out of curiosity, did you ever notice other shells crashing? I’ve seen terminal gui apps crash, or some nasty shell programs or functions or such, but I don’t remember ever seeing a shell itself crashing, ever since csh and tcsh in the nineties.
Maybe I saw it - e.g. choking on too big an input flood - but I don’t think I saw a shell crash. I’m now curious, what have I missed?
I have had bad bugs but no crashes. Thing with fish is I never expected this level of stability as it clearly had much less resources than other shells when I switched.
I am so glad that they made the code FUN! I routinely install fish on ALL my systems. I am addicted to autocompletions and abbr commands. I also like that it’s in a memory safe language. One quick question, when I compile it on Amazon Linux 2023 there are a 50 character when using the aws web console. Has anyone seen this and if so is there a fix?
there are a 50 character when using the aws web console
I think you accidentally a word, friend. But I’m guessing it’s that the aws web console doesn’t support an ANSI escape we issue as part of our interop with the terminal emulator. A proper issue on GitHub with a screenshot would be appreciated!
It’s a static html site, source code is on GitHub. If you want to manually write an atom feed for the entries and we just have to remember to add to it each time, I guess you could try opening a PR?
That one was less abrupt in nature; it was really just first allowing C++ code then slowly (and I mean slowly) embracing RAII, generics, and other C++ niceties. You never really can fully take the C out of C++!
I think the bigger other rewrite we did was actually going from autoconf to cmake.
This is all pretty encouraging and matches my experience at $WORK: converting code in chunks while keeping the system functional is very doable with minimal problems.
My only difference is opting into hand-written cxx FFI layer:
it rarely is difficult but saves lots of headaches
with its simplicity. Also it is temporary so is very pleasant to drop.
We had to fork cxx to add some features though (and I still need to finish some).
I’m one of the core team; we didn’t want to get into it too much in the linked post but we definitely did a lot of C abi to avoid (auto)cxx issues. It was more work but at least it is very transparent and you can reason about what is happening rather fight layers of machine-generated code and need to invent new ways to marshal lifetimes across abi boundaries. As a result, some of the rust code is actually still structured in a way that’s intentionally amenable to using across a C abi and we’re slowly making it more idiomatic.
Our own forks of autocxx and cxx are part of the fish-shell org on GitHub, though they’re purely historical at this point (kept around both for posterity and so we can properly git bisect builds across multiple generations of code).
I’m very curious what are you doing with panics (not obvious unwraps)? We found them creeping out in unexpected places, which forces us to catch_unwind a lot. Which is not nice and is a reasonable critique from reliability perspective.
Well, we’re using very few third-party dependencies and don’t believe we have any panics other than those that might arise from holding the thing wrong, where we’d prefer to panic than continue in an inconsistent state in all cases. We didn’t mention it in the article but restructuring the code to properly pass along errors was a big part of the undertaking. (The old code also had error handling but it was often more locally scoped.)
We intentionally panic internally on a few libc failures - but we always have if we’re not able to put the system in the state we expect to be able to. Things like memory allocation failures, the os returning unexpected errors to syscalls, etc are all places where both the old and the new codebases panic (with appropriate error messages when and where possible) the same. The rust standard library documents where its own functions might panic (and generally offers non-panicking alternatives that are less ergonomic to use) and we try to make the right choices.
We do run into panics from time to time, but it’s always because of an actual bug (but of course that depends on your personal philosophy; ours is that it’s preferable to panic and get a reproducer then fix or even rearchitect as needed to avoid that panic). We wouldn’t want to catch those and continue!
FWIW there are a few projects which did something similar previously, librsvg is probably the most famous one, the maintainer (Frederico Mena-Quintero) did writeups across 2 or 3 different blogs, and notably it did releases of the in-flight / partially converted codebases.
However as it’s a library which exposes a C API and was converted from C there would obviously be divergences in both requirements and experience.
I guess I can plug our open-source git-based secrets management solution here, which is made of command-line tool (for managing secrets) and an api that integrates directly into your application (for retrieving them dynamically at runtime). It’s a KISS secrets management system approach that eschews having servers to provide secrets to your projects at runtime; the implementation is standardized and has been ported to different languages and frameworks.
The approach is meant to be dependency-free and flexible in terms of how you deploy secrets into dev/staging/prod environments; we’ve seen people using it in creative ways.
Confused. What am I supposed to under from that screenshot? Yes, fish is not zsh. I don’t see a single example of interactivity there. All I see are commands and their output.
Many fish users don’t use ls at all. Myself included. I can navigate and enter the directories interactively with tab completion. Ls becomes unnecessary.
Although, it gives me pause that it’s not enabled by default when the config includes programs.fish.enable = true;. Presumably there’s a good reason for that.
update: trying it now, seems to work great. thanks again for the tip!
Functions wouldn’t work in bash with direnv anyway: the underlying issue is a difference in workflow with that colleague (who seemingly doesn’t use direnv). The solution is to use scripts instead of functions.
Is this just a matter of nix providing some script that can be sourced by fish setting up certain global state and maybe chrooted into a certain path? Are there things fish doesn’t support that stops it from being an option or is it just lack of an upstream fish version of the script?
nix-shell is also used for debugging package builds, with genericBuild, build phases, etc… Those won’t work on anything but Bash due to the builder being “highly advanced” bash code.
If it’s only to get a few environment variables in your shell, direnv works well.
Lovely to see the changes and improvements. I keep eyeing Fish to replace my Zsh setup, but worry about losing out on years of muscle memory for POSIX-ish shell syntax.
In other news, I wish the blog has an RSS feed, I’d like to keep up to date with new releases to read about features etc…
As a prime example of this, a while back they added the ability to do FOO="blah" some-command instead of the previous version where you need to prefix with env. This alone resolved something like 90% of the ways my daily use of fish diverged from what I would have written in bash or zsh.
I colleague of mine recently switched. That surprised me because he had quite some zsh setup and as far as I know fish does not really offer more feature-wise. He told me that he likes fish because it is more „snappy“.
Out of the box, Fish has far more features than Zsh. It doesn’t offer anything else feature-wise if you install all the Zsh modules that were created to implement Fish features, but they’re fiddly and often just don’t work as well. If you want Zsh like Fish, just use Fish.
I agree. The point is if you already invested the time to set up zsh with all kinds of modules, then switching to fish is not much of an improvement. So I don’t recommend fish to zsh power users.
That said, I have now the anecdotal evidence from one person that fish was still worth switching.
It’s far less janky than zsh, mostly because you have less user-authored code (users in general, not you specifically). I don’t touch my config for months on end and things just keep humming along well.
I don’t touch my config for months on end and things just keep humming along well.
I don’t touch my Zsh config for years on end and likewise. On the other hand, I imagine my Zsh config took longer to write than your Fish config, though.
My brother in christ, please just let me switch on strings.
The word “just” is sitting on top of a lot of complexity in this statement.
Pardon my lack of zig experience but I wouldnt expect any reference type to work in switch in a systems language (similar to C)
I would count rust as a systems language and it works in magch there (not just for &str but any reference).
Any constant reference.
Do you want equivalent but differently encoded unicode strings to be considered equal?
No, just raw byte-for-byte comparison.
None of the workarounds in the article seem to address that either.
Do you think that not allowing users to switch on strings will avoid canonicalisation issues somehow?
Can users define their own string type and make switch work on it?
would it make sense to allow an optional comparator to the switch statement? e.g.
switch (mystr, &std.mem.equal) { ... }I had been imagining something like the ability to extend
switchto operate on any aggregate type that has a hash-function definition available, but having a comparator generalizes that, and even supports some interesting use-cases like canonicalization-in-comparison (as @andrewrk asked about) as well as even fancier things like edit distance matching!I find this idea quite compelling…
All the best,
-HG
Sounds like pattern matching
NB: I specifically don’t want to get into any political discussion, merely a statistical one.
That said, surely with 50% of the respondents self-reporting as belonging to the LGBTQIA+ communit{y,ies} this indicates a statistically significant selection bias in the targeted users? I’m not complaining about (in)equity but I am questioning whether this fairly “purely demographic” question ends up indicating that the answers for the remaining questions may not hold for “the average rust user” as rust has clearly escaped niche status and therefore one would assume that demographic distribution of users skews somewhat closer to the baseline distribution of (system or backend) developers in general, where you’d expect that number to be smaller (though by how much apparently varies).
Lest one consider this is still an attempt to disguise rage politics, I want to give a specific point: I think the avenues where this survey was being suggested might skew to a specific demographic (not in terms of gender identity specifically, but also wrt nature of $work, age, experience, etc) and exclude a good chunk of potentially relevant (to rust maintainers) users. With a sample size of one: I saw (and believe I participated in?) the 2023 survey but I didn’t hear about the 2024 survey despite being on the rust github and associated blogs “all the time”, but one likely explanation is that I quit reddit (and therefore r/rust) in 2023. And as another point, I know the most tech-active mastodon nodes just happen to be most active on the lgbtq+ mastodon servers.
Not all questions were answered by all participants of the survey. Note that there were 1110 total responses to that question. The interpretation should be that 50% of the people who consider themselves to be a part of an underrepresented and/or marginalized community report as being LGBTQIA+.
I think the blog post does a very good job at explaining that:
I thought the survey was well-publicized — I found out about it by following the official Rust blog in my RSS reader. I acknowledge you may not be trying to disguise rage politics, but you do seem to be implying very strongly that there was some sort of an attempt to bias the survey towards LGBTQIA+ participants without understanding the basic results of the survey.
Even with that adjustment, one point that I find very interesting is that 15.5% times 35.8% (people who answered “trans”) works out to an overall fraction of 5.5%. That’s 10x higher than the general population.
As a trans person who knows a lot of trans people who use Rust, this doesn’t entirely surprise me, but it’s slightly mind-blowing and also probably at least somewhat affected by selection bias.
Thanks for pointing out that possible statistical explanation for the discrepancy I noted; it potentially answers my question in full.
(I don’t see how any part of my post could be interpreted as accusing the rust org to try to intentionally skew towards certain users; for the record I hold the maintainers in very high regard. I thought it was clear I was only reaching for possible factors that could explain the demographics of the respondents. If I said something along the lines of “they clearly should have used xxx as a venue”, then maybe; I only indicated my activity on certain hubs as a signal of how keyed in to the language I personally was, for the benefit of readers of my comment. And my comment certainly should not be read to imply that I have any problems with any demographic group, because it certainly doesn’t say that and I certainly don’t.)
I wasn’t saying that you were accusing the Rust maintainers of skewing the data, just that your comment sounded like it was hinting at skewing the data towards certain demographic without fully appreciating the results.
That said, I have no reason to doubt your goodwill, I share your high regard for the maintainers, and I’m glad there was a satisfactory explanation for your question!
To just put the numbers together, 7310 completed surveys, 1110 * 54.1% users reporting as LGBTQ, is 8% of respondents choosing to report as LGBTQ. Combined with the 10% who preferred not to say that’s 8 to 18% of respondents are LGBTQ, I’d guess somewhere towards the lower end of that range.
Lovely read and amazing perseverance and persistence… but the conclusion/benchmark missed the biggest wins. No mainstream linker supports incremental linking (not even mold) and that is where the biggest gains lie. With a real world codebase, for example, linking fish-shell takes 1-3 min per binary (there are three of them) in release mode; this applies even if you change a single character in the core codebase. While normally the edit-debug loop is done in debug mode, when you are editing for performance you want release builds (and we do that often). Prior to switching to rust (and static linking), our C++ codebase would (dynamically) link the binaries in a couple of seconds at most.
Am I the only one whose browser froze trying to scroll the page initially? (Chrome on Windows)
I had similar behavior with Edge on Windows. In my case, my browser and screen reader froze when I tried to read the page.
The secret is that one says third party projects need to live in your monorepo. If you use a sane language with support for pinning, the only time you have third party source code you need to include in your repo is when you’ve forked it - i.e. when it’s no longer third party by definition.
Things like hosting your own module/package/nuget/cargo/whatever cache servers in-house (optionally restricted only to package/version combos you’ve vetted for license and security compliance) takes you the rest of the way there.
This is just moving from the unix-style cli to the old win32-style cli. Except Microsoft gave up and ended up embracing ANSI escapes and CSIs (some time after WSL landed) to support the ever-growing corpus of apps specifically targeting terminal emulators.
In the Win32 console model, apps emitted only text to stdout and used the console handle to do everything else from setting attributes to switching between the full screen buffers. And the open source world made fun of them for it.
I wouldn’t do that, it isn’t like the unix terminal design is exactly ideal either.
I don’t think the unix design is as offensive on the output side as the input though; if you’re just writing a program that wants to output some colorful text, it isn’t all bad. Just remember to check isatty and the command line flag that overrides that before adding the color sequences!
On the input side though, yikes, what a pile of disgusting hacks. You have to do a timeout check on the esc char! And woe be upon you if you need to query the terminal for something, since now that’s both input and output sequences! Worth noting a lot of the newer extended terminal emulators are introducing new features to change this stuff, but they’re tied by the need to be compatible with the large existing ecosystem of applications.
There’s also the separate question of api vs underling data too. In the underlying data, one stream does make synchronization and redirection and other things easier. But then in the api you might want it separate anyway, to make supporting
--color=autoand--color=offeasier. Stripping escape sequences from the output isn’t awful though, so you could just always write them in your code and have the library function just does that.Nevertheless, I’m not a fan of making fun of designs… there’s often fair reasons for differences that you can learn from.
If you generalise to that level, well, duh. The argument that in-band signalling UI is harmful in the same way smashing strings together versus prepared statements is not a difficult one.
Microsoft didn’t “give up” – it was a no brainer for them to throw a few idle headcounts on writing an emulator for their WSL trajectory and getting more developers into VScode. They’re otherwise busy adding AI keys and ads in the UI so perhaps their sensibilities is not something to use as ‘appeal to authority’ or inspiration.
I am intimately familiar with the inner workings of the different console layers in the MS ecosystem and their reasons for being – from the days of conio.h, int 21h, mode con codepage prepare and onwards. Their powershell model is much closer in comparison but the power they had there came specifically from them having something of a functional IPC system to work with. Linux/BSD isn’t so lucky.
Isn’t a modern Thunderbolt cable (using USB-C phys) a superset of the basic USB-C cable?
Maybe it’s a “this old 10/100 network hardware bugs out in this configuration unless you use a 10Mbit switch or a managed switch with the port forced to 10Mbit operation” situation.
This list of deficiencies reads like a slightly obscured, writ-large version of “don’t roll your own crypto”.
Ah, but don’t you know, they “aren’t” rolling their own crypto, per their FAQ.
heavily rolls eyes
I like libsodium. It’s a great library of cryptography algorithms.
It doesn’t come, batteries included, with a full protocol for end-to-end encryption built-in. And so anyone who uses libsodium for e2ee is necessarily rolling their own crypto.
I’ve only heard of Session in passing over the years. This result is not surprising.
i initially thought you were being overzealous, until i read the first paragraph of the article
using libsodium to implement the documented signal protocol, i think would be fine. it does have risks, and should have some review before relying on it, but i wouldn’t really call that “rolling your own crypto”. and having a clean-room re-implementation would probably be good for the ecosystem
…but that’s not what they’re doing. they’re implementing their own protocol, and a cursory glance at their reasoning suggests that they want a decentralized messenger and care about security as an afterthought. which would be fine if they branded it that way, and not as an alternative to signal
This may be a little off topic, but I dislike the phrase “don’t roll your own crypto”.
Don’t roll your own crypto is generally a term which in itself is very ambiguous.
I’ve seen this phrase being thrown around when people just use gnutls in C vs people implementing some hash algorithm themselves. One I find very valid while the other one is an area where I would just use libsodium.
There are so many layers in crypto where you can apply the phrase that I find refuting (their claims with) this phrase in itself is meaningless unless you know what the authors intended. In this case it may as well be claims regarding the resistance against certain side channel attacks.
I’ve always asked myself how I can identify the moment I arrive at a skill level where I’m allowed to “roll my own crypto” depending on each possible interpretation people are using.
edit: added intended missing meaning in (…)
Absolutely. And the advice, taken to its logical extreme, would result in zero cryptography ever being developed.
It’s supposed to be along the same lines as advice given from lawyers to their kids that are considering a career in law. They say, “Don’t be a lawyer,” and if their kid isn’t dissuaded and can argue why they’d succeed as a lawyer, then maybe they should be one.
“Don’t roll your own crypto” is along the same lines. I currently do it professionally, but I also have a lot of oversight into the work I do to ensure it’s correct. Detailed specifications, machine-assisted proofs, peer review, you name it. I never start with code; I always start with “here’s why I’m doing this at all” (which includes closely related ideas and why they do not work for the problem I’m solving) and a threat model for my specific solution.
It can take months, or even years, to get a new cryptography design vetted and released with the appropriate amount of safety.
When it comes to cryptography protocol design, the greatest adversary is often your own ego.
I always read the advice as an incomplete sentence, which ends with “unless you know what you’re doing”, which is coincidentally like other safety advice, right? “This thing that you’re about to do is risky and dangerous unless you know how to do it, and in some cases, even if you do. Avoid doing it if you can. Exercise caution and care otherwise.” No?
I always viewed it as “don’t ship your own” - feel free to roll your own to play around with, but be cautious and get some review before putting it into production.
One piece of advice I’ve heard is: Before trying to build crypto, learn how to break crypto. Cryptopals is a good resource for that. It’s mindbending to learn about all the weird ways that crypto can fall apart.
At least one of my online friends agrees.
I think it’s more like, don’t roll your own crypto: don’t do it by yourself, collaborate with other experts, get lots of review from many angles.
I remember many moons ago that an expert in security and crypto actually published a list of cryptographic choices that should be your default. I wonder if this rings a bell to someone, it would be nice to recover that document, publish it here and see what this community would say in terms of freshen it up.
I might be wrong, but I think in the beginning the meaning of the phrase “don’t roll your own crypto” mean “do not try to come up with cryptographic algorithms on your own; use something tested and done by someone who know what they are doing”. I think the best way to describe what Soatok is putting forward is “don’t skip the default practices of security” or “don’t wave away cryptographic defaults in name of a watered down threat model”.
But maybe I am too off?
You might be thinking of “cryptographic right answers” from ’tptacek (2018 version, 2024 post-quantum version)
YES!!! You found it! Thank you @zimpenfish!
There’s also What To Use Instead of PGP from the very blog this Lobsters thread is about.
It was also posted on Lobsters.
Maybe I’m paranoid, but it reads to me like a plausibly deniable honeypot.
I think that’s a very reasonable concern. Particularly in light of the very first issue @soatok cites: the removal of PFS from the protocol. I’m on record as being skeptical of the “just use signal” advice that seems frequently deployed as a though-terminating measure in discussions about encrypted communication, but if I wanted to make something that was like signal but really a deniable honeypot, Session makes the same choices I would. It seems like a downgrade from signal in every important way.
Unrelated: the “Ad-blocker not detected” message at the bottom of the post made me laugh quite a bit. I use some tracker-blocking extensions (and browser configs) but I don’t run a real ad blocker in this browser. But many sites give me an “ad blocker detected” message and claim I need to turn off my non-existent ad blocker to support them. This site telling me I’m not running enough of one struck me as very funny.
Sure, its plausible.
But I find basically every time Soatok (or any security researcher) exposes any application that advertises itself as “secure/private” on the box, for their glaring bad practices, people (myself included) immediately go to “this is so stupid it has to be a honeypot”.
Are they all honeypots? (genuinely, maybe yes), or is it just stupidity?
i used to think that people sending a knockoff paypal payment link from a TLD i’ve never heard of was an obvious scam
then i tried to buy a monitor from hewlett packard via customer support, and i found out who these scammers are imitating
I would posit stupidity. Previous honeypots that weren’t takeovers of server operators have been somewhat targeted: Anom required a buy-in of the phone (as evidence you’re a criminal), Playpen required you be a pedophile (or at least, hanging out online with pedophiles) to be caught in the net, Hansa was a drug market, etc. Creating a general-purpose backdoored app to en masse catch criminals seems to cast quite a wide net when the same arrest portfolio can probably be gathered by doing the same thing to Twitter DMs with a backdoor and a secret court order. I wouldn’t put it past law enforcement but it seems like a mega hassle vs. targeted honeypots and backdoors.
If it were a honeypot (or backdoor), it’s certainly too much hassle for legitimate law enforcement purposes like the ones you described. You’d want this for someone you couldn’t reach through normal court (even a secret rubberstamp like FISA) channels.
This would be more like something you’d use for getting information from a group operating under some legal regime that’s not friendly to you gathering that information. Getting it in place, then convincing the group you were interested in to migrate off, say, Telegram, might be one approach.
The interesting thing in this case (IMO) here is that the fork removes things that were:
and without articulating the upside to their removal very persuasively. Sure, stupidity is always a possibility. But it feels more like they want to add some features that they don’t want to talk about. On the less-nefarious end of that spectrum, I could imagine that it is as simple as supporting devices that don’t work with the upstream, but that they don’t want to discuss in public. It’s also easy to imagine wanting to support some middle scanner-type box on a corporate network that PFS would break. But it could also be something like being able to read messages from a device where you can maintain passive traffic capture/analysis but can’t (or would prefer not to) maintain an ongoing endpoint compromise without detection. e.g. You have compromised a foreign telco and can pull long term key material off a device when its owner stays in your hotel, but you can’t or won’t leave anything running on there because the risk of detection would then be too high.
That’s just all speculation about when it might serve someone with an effectively unlimited budget to do something like this. Stupidity is certainly more common than such scenarios.
Hence “plausibly deniable.”
Only the first bit could charitably be attributed to “don’t roll your own crypto”. The rest was just obtuse idiocy or malevolence. Call the library-provided “encrypt this chunk with symmetric encryption using this key” then providing a public key.. that’s not about rolling your own crypto.
From 2022, but I don’t think it’s been discussed here before.
Really cool and it’s nice how easy this is to pull off under Linux in user space. I can’t begin to think how many drivers I believe I’d need to write to do this on Windows.
One nitpick with regards to the invocation of Shannon’s law: the 1.5Mbps isn’t the max theoretical channel limit; that’s the max digital rate limit of the underlying digital tunnel OP chose (raw pcm, 16-bit, 48MHz, etc).
You could certainly do much better performance-wise and blow past the so-called 100% limit named in the post with a modern modulation scheme instead of the basic differential Manchester/bi-phase mark coding used by audio standards to define the modulation at the physical layer over toslink fiber. I think there are plenty of quadrature amplitude modulation open source libraries you could stick in there as a start, but you’d have to read/write raw signals to the toslink channel, not going through arecord and aplay.
I’ve used fish as my main shell for something like 13 years and while I have no real opinons of a rust rewrite. I’ve been pleasently surprised by the stability of the shell, since its been a small hobby project of someone I dont know, I’ve granted it a crash and bug allowance. However, fish has never crashed on me nor have I experienced any bugs that made me drop it even for a day.
If rewriting in rust is what the team thinks is right I trust you and wish you keep your fun in making it. I’ve loved using it and I hope you don’t lose interest any time soon.
Thanks for your support and patronage! We don’t generally release beta builds (our philosophy is to keep
masteras stable as the last release or better) but this has been an exceptional process and we’re hoping that with a beta release we can catch any real-world crashes we know must have been introduced but not yet caught, thanks to the laws of statistical probability. So if fish does crash for you, please do let us know and rest assured that we are doing our best to make that as unlikely as possible.Out of curiosity, did you ever notice other shells crashing? I’ve seen terminal gui apps crash, or some nasty shell programs or functions or such, but I don’t remember ever seeing a shell itself crashing, ever since
cshandtcshin the nineties.Maybe I saw it - e.g. choking on too big an input flood - but I don’t think I saw a shell crash. I’m now curious, what have I missed?
I have had bad bugs but no crashes. Thing with fish is I never expected this level of stability as it clearly had much less resources than other shells when I switched.
I am so glad that they made the code FUN! I routinely install fish on ALL my systems. I am addicted to autocompletions and abbr commands. I also like that it’s in a memory safe language. One quick question, when I compile it on Amazon Linux 2023 there are a 50 character when using the aws web console. Has anyone seen this and if so is there a fix?
I think you accidentally a word, friend. But I’m guessing it’s that the aws web console doesn’t support an ANSI escape we issue as part of our interop with the terminal emulator. A proper issue on GitHub with a screenshot would be appreciated!
If anybody from the fish team reads these comments, could I please ask for an Atom/RSS feed for the blog? Thank you :-)
One of our co-maintainers figured it out; here you are: https://fishshell.com/blog/feed.xml
Thank you :-)
It’s a static html site, source code is on GitHub. If you want to manually write an atom feed for the entries and we just have to remember to add to it each time, I guess you could try opening a PR?
https://github.com/fish-shell/fish-site
Could be a script to generate it in a pre-push hook. Will think about it if I pop the yak stack for it.
It’s a Jekyll site so you could use a plugin like jekyll-feed.
Crazy there was a rewrite to C++ from C in the history too.
If the goal is concurrency & performance, I am in as a long-time Fish user.
That one was less abrupt in nature; it was really just first allowing C++ code then slowly (and I mean slowly) embracing RAII, generics, and other C++ niceties. You never really can fully take the C out of C++!
I think the bigger other rewrite we did was actually going from autoconf to cmake.
Many thanks for writing (and posting) this up.
This is all pretty encouraging and matches my experience at $WORK: converting code in chunks while keeping the system functional is very doable with minimal problems.
My only difference is opting into hand-written cxx FFI layer: it rarely is difficult but saves lots of headaches with its simplicity. Also it is temporary so is very pleasant to drop.
We had to fork cxx to add some features though (and I still need to finish some).
PS Switched to fish a year ago
Welcome to fish, friend!
I’m one of the core team; we didn’t want to get into it too much in the linked post but we definitely did a lot of C abi to avoid (auto)cxx issues. It was more work but at least it is very transparent and you can reason about what is happening rather fight layers of machine-generated code and need to invent new ways to marshal lifetimes across abi boundaries. As a result, some of the rust code is actually still structured in a way that’s intentionally amenable to using across a C abi and we’re slowly making it more idiomatic.
Our own forks of autocxx and cxx are part of the fish-shell org on GitHub, though they’re purely historical at this point (kept around both for posterity and so we can properly
git bisectbuilds across multiple generations of code).I’m very curious what are you doing with panics (not obvious unwraps)? We found them creeping out in unexpected places, which forces us to catch_unwind a lot. Which is not nice and is a reasonable critique from reliability perspective.
Well, we’re using very few third-party dependencies and don’t believe we have any panics other than those that might arise from holding the thing wrong, where we’d prefer to panic than continue in an inconsistent state in all cases. We didn’t mention it in the article but restructuring the code to properly pass along errors was a big part of the undertaking. (The old code also had error handling but it was often more locally scoped.)
We intentionally panic internally on a few libc failures - but we always have if we’re not able to put the system in the state we expect to be able to. Things like memory allocation failures, the os returning unexpected errors to syscalls, etc are all places where both the old and the new codebases panic (with appropriate error messages when and where possible) the same. The rust standard library documents where its own functions might panic (and generally offers non-panicking alternatives that are less ergonomic to use) and we try to make the right choices.
We do run into panics from time to time, but it’s always because of an actual bug (but of course that depends on your personal philosophy; ours is that it’s preferable to panic and get a reproducer then fix or even rearchitect as needed to avoid that panic). We wouldn’t want to catch those and continue!
FWIW there are a few projects which did something similar previously, librsvg is probably the most famous one, the maintainer (Frederico Mena-Quintero) did writeups across 2 or 3 different blogs, and notably it did releases of the in-flight / partially converted codebases.
However as it’s a library which exposes a C API and was converted from C there would obviously be divergences in both requirements and experience.
I guess I can plug our open-source git-based secrets management solution here, which is made of command-line tool (for managing secrets) and an api that integrates directly into your application (for retrieving them dynamically at runtime). It’s a KISS secrets management system approach that eschews having servers to provide secrets to your projects at runtime; the implementation is standardized and has been ported to different languages and frameworks.
Here’s the rust version: https://github.com/neosmart/securestore-rs and here’s a writeup that’s largely been superseded by the expanded README: https://neosmart.net/blog/securestore-open-secrets-format/
The approach is meant to be dependency-free and flexible in terms of how you deploy secrets into dev/staging/prod environments; we’ve seen people using it in creative ways.
Previously, on Lobsters:
That’s a lot of previouslies!
My contrarian opinion https://lobste.rs/s/ldkfdg/sudon_t
With all respect to fish(1) … its not a good interactive shell.
Example: https://pbs.twimg.com/media/GfRKRSGWEAAlRuR?format=jpg&name=medium
Not to mention lack of POSIX syntax for while(1) or for(1) loops …
The zsh(1) shell is way more usable for interactive work:
https://vermaden.wordpress.com/2021/09/19/ghost-in-the-shell-part-7-zsh-setup/
We are lots of people who’ve enjoyed using Fish shell for years. If you have another favourite shell, great for you.
“it’s not a different shell that has different syntax” is a pretty poor argument. It requires learning.
Fish has its own paradigms and workflows. But if you really want, recent versions of fish have a one-liner you can use to enable support for
!!Good to hear that.
Maybe in the future POSIX syntax and
!$will also be supported via various workarounds … or implemented.Confused. What am I supposed to under from that screenshot? Yes, fish is not zsh. I don’t see a single example of interactivity there. All I see are commands and their output.
Many fish users don’t use ls at all. Myself included. I can navigate and enter the directories interactively with tab completion. Ls becomes unnecessary.
As a NixOS user I wish it were possible to get fish inside nix-shell.
I can say that with Direnv, I’ve rarely needed raw
nix-shell. What kind of use case do you have in mind where that wouldn’t work?Also I assume you’ve heard of any-nix-shell?
I tried direnv today at your suggestion
I hadn’t heard of it, thanks! Will give it a try.
Although, it gives me pause that it’s not enabled by default when the config includes
programs.fish.enable = true;. Presumably there’s a good reason for that.update: trying it now, seems to work great. thanks again for the tip!
final update: very happy with my setup now
Works fine until your coworker start injecting Bash-only functions into the dev shell for some reason 🤦
Functions wouldn’t work in bash with direnv anyway: the underlying issue is a difference in workflow with that colleague (who seemingly doesn’t use direnv). The solution is to use scripts instead of functions.
Is this just a matter of nix providing some script that can be sourced by fish setting up certain global state and maybe chrooted into a certain path? Are there things fish doesn’t support that stops it from being an option or is it just lack of an upstream fish version of the script?
nix-shellis also used for debugging package builds, withgenericBuild, build phases, etc… Those won’t work on anything but Bash due to the builder being “highly advanced” bash code.If it’s only to get a few environment variables in your shell,
direnvworks well.Lovely to see the changes and improvements. I keep eyeing Fish to replace my Zsh setup, but worry about losing out on years of muscle memory for POSIX-ish shell syntax.
In other news, I wish the blog has an RSS feed, I’d like to keep up to date with new releases to read about features etc…
We’ve made great progress in supporting posix syntax and features that don’t clash outright with fish. You should give it another try!
As a prime example of this, a while back they added the ability to do
FOO="blah" some-commandinstead of the previous version where you need to prefix withenv. This alone resolved something like 90% of the ways my daily use of fish diverged from what I would have written in bash or zsh.I colleague of mine recently switched. That surprised me because he had quite some zsh setup and as far as I know fish does not really offer more feature-wise. He told me that he likes fish because it is more „snappy“.
Out of the box, Fish has far more features than Zsh. It doesn’t offer anything else feature-wise if you install all the Zsh modules that were created to implement Fish features, but they’re fiddly and often just don’t work as well. If you want Zsh like Fish, just use Fish.
I agree. The point is if you already invested the time to set up zsh with all kinds of modules, then switching to fish is not much of an improvement. So I don’t recommend fish to zsh power users.
That said, I have now the anecdotal evidence from one person that fish was still worth switching.
It’s far less janky than zsh, mostly because you have less user-authored code (users in general, not you specifically). I don’t touch my config for months on end and things just keep humming along well.
I don’t touch my Zsh config for years on end and likewise. On the other hand, I imagine my Zsh config took longer to write than your Fish config, though.
it is still an improvement because you end up with less shell code