Wow, this is exactly the reaction I kinda imagined when I saw paperwm.
Before doing the switch from awesome to paperwm I found Niri and decided that I’ll move directly to wayland+Niri instead. Though I kinda haven’t had the guts to leave all my X helper scripts behind and actually move to wayland yet…
I guess this is what I needed to actually seriously consider switching to wayland soon. Thanks!
I found paperwm first, and it just really didn’t work for me, leading to me ignoring niri for a fairly significant amount of time by association. This post gave me the nudge to actually try it out, and I’m very pleasantly surprised!
Going to be switching to it fully for a while to really get a feel for it, and then decide if I’ll stay definitively.
Running smoothly! (+ decoration handling, scaling, shortcut management…)
Using PaperWM felt like replacing my touchpad with a cheese grater comparatively (I’m typically a labwc user).
With PaperWM, you get all the integration that Gnome provides, mainly in terms of managing hardware devices. You can provide all of this for Niri, and I have (originally for Sway, but it was trivial to bring my Sway configuration over to niri), but there’s no turnkey solution.
On the other hand, niri is much faster and smoother than PaperWM, and less buggy (because PaperWM is forced to keep up with Gnome Shell compatibility and interaction with other extensions). I also find that it handles the interaction between workspaces and multiple displays better than PaperWM.
Generally, I haven’t had many complaints with PaperWM (knocks on wood) or found it to be buggy. I do worry about the keeping up with GNOME – right now development with PaperWM seems active enough, but I can easily see a scenario where the developers move on and it stops working with newer versions of GNOME. If that day comes, I guess I’ll definitely be moving over to niri whether I want to or not.
Simplex sounds interesting. Giving up on having any kind of identifier that a human can read remember and type makes for interesting UX challenges but if this can be handled by address book management it may provide a much better way of exchanging encryption keys.
I am interested in review of the crypto and privacy properties. There doesn’t seem to be anything on the IACR eprint server. Has nobody looked at this yet?
The default shell on every major Unix, Linux, BDS and even MacOS hasn’t had a single meaningful new feature for over 25 years!
Good. GOOD.
I’m seriously considering trying Oils, but purely for the footgun-elimination aspect. Having a feature-complete, stable, default shell is most definitely a feature as far as I’m concerned.
Very underappreciated argument, thanks for bringing it up!
I think it’s a question of how much convenience (features) you’d want on a modern shell.
IMO the “feature-complete” POSIX Standard is not enough. And the Bash features make everything only confusing/are not really “complete”.
If only GitLab introduced seamless Pijul support! That is, make sure that both Pijul and Git clients can interact with repos in a sensible way. Then I could learn the ins and outs of Pijul with an easy fallback in case of catastrophic usability failure.
Yeah I guess as long as there is no open hosting solution the gate is kinda closed for wide adoption :/
tbh I haven’t looked into it’s ecosystem for a long time now. I wonder how pijul vs jujutsu would compare/if a pijul “backend” could be supported by jujutsu.
Not sure. The question is who is the attacker. If it’s state agency, then definitely not.
If a state agency is targeting you, you probably don’t stand a chance. But given how state leaders are shifting more and more to the fascist extremes (left and right…), complete citizen observation isn’t just a topic for china anymore.
So not wanting any data on the server is IMO a very valid concern, because that makes “observe everyone” much harder.
I’m attracted to purism. I need innovation plus an application to keep reading. I need something that’s thoroughly thought through to really like it. [..] For an example Pijul satisfies almost all. But git is kinda “good enough” (for me at least). The benefits don’t seem to really justify replacing a whole ecosystem of how we manage code. So there isn’t really an application for me to start using it. I’d definitely use it if others did, but I wouldn’t push it.
I followed the same train of thought and decided to go with Jujutsu (jj). Like pijul, it also has new, strong design ideas – but it is less radical. Unlike pijul, it is compatible with git repositories and forges, which means that it has a much easier path to adoption. (My application is “much better experience than git when rewriting histories in preparation for one or several github PRs”.)
Yes true! I have it in my backlog to take another look (didn’t get it last time i tried it, I didn’t realize I should change the workflow :)).
I wonder if the longterm goal of Jujutsu is to introduce an alternative backend which is more “native” to Jujutsu. In that case it would be taking “the long road” very similarly to Oils. Reimplement, then improve.
I have the feeling this is a currently very liked approach. E.g. coreutils is one other example (thought this is more about the language/development ergonomics than features/usage economics).
I want to say thanks to @dgl for reporting this, and this article is also expertly written.
I also have to say for me personally its quite embarrassing because as I told @dgl when he reported this: I’ve studied his work before and made it a note to test Ghostty against his past discoveries prior to release. And I… quite simply forgot. I didn’t make an issue for myself so it slipped away and here I am with egg on my face. I’m sorry!
But, I appreciate @dgl for the security report, reviewing the fix, and continuing to be an active user of the terminal. I also shared with him some broader thoughts on terminal security in general. Addressing terminal security in a more fundamental way is one of the first proposals I want to make regarding terminal innovation.
My thinking is still too early and under-researched for a formal proposal. But my general feeling having built a terminal over the past 2 years is that the security surrounding escape sequences is fundamentally flawed and poking one by one at these sequences to try to make them individually safe doesn’t feel like the right long term solution.
The surface area is too large and the complexity of some of the newer sequences too high (i.e. Kitty Graphics) to be confident in secure implementations. DoS is far too easy with terminals (Ghostty has a handful of known DoS attacks, but so does pretty much every other terminal I know of). And some legacy sequences are just kind of shocking to have immediately available. For example, DECCOLM is available in macOS Terminal.app. If you issue a DECCOLM (CSI ? 3 h I believe), Terminal.app will physically resize the window and lock it to 132 columns. You can very easily crash Terminal.app at anytime by sending enough of these (a DoS attack). There are many more.
Part of my thinking is trying to design a mechanism that can effectively create something akin to either CPU protection rings or OpenBSD’s pledge() syscall. Whatever the mechanism, the general idea is: reduction of capability.
For example, it makes sense for a shell to be extremely powerful. It’s literally a code execution device. However, it doesn’t make much sense for cat to be able to execute arbitrary escape sequences (which it does today in any terminal). Whether it is the shell or cat itself, something should be able to tell a terminal: “hey, only execute some subset of escape sequences (or none at all).” For cat, that might be none or perhaps just SGR sequences (styles, colors, etc.). The insecurity of cat-ing or tail-ing any form of data that could contain user-generated data is hopefully well known at this point…
As I said before, this thinking is all still very raw and I don’t have anything concrete to proposal. I want to share this because I want folks to know that I’m thinking about it, and perhaps others may be interested in thinking about it with me… if so, please contact me. I’ve already shared this line of thinking with @dgl as well.
I wanted to send you a mail for some time but here we go :)
@andyc, author of Oils has designed the currently very simple FANOS Protocol. It itself has nothing to do with escape sequences. It’s merely for seperating inputs & outputs of commands. Ie the terminal sends the shell EVAL cat bla and 3 file descriptors, the shell executes cat bla and uses the fds for STDIN/STDOUT/STDERR. The terminal has the other ends of the fds open and talks to/reads from cat directly.
Why is this relevant? For your example there’s currently no way for the terminal to know who sent what output (e.g. if I run something like {sleep 0.1; printf 'EVIL ESCAPE';}& cat huge-file). FANOS would pass the control for merging the output streams over to the terminal. This solves many issues where currently workarounds need to be used, like overly complicated PROMPT_COMMAND, etc. And it could be a cornerstone to also improve security, interactive features, etc. as the terminal suddenly has much more knowledge.
IMO this is the kind of protocol which should’ve been developed 30 years ago.
If you’re interested, we’d be super glad if you would join the #shell-gui channel on the Oils Zulip chat for feedback & discussions. The protocol is also super raw (only has EVAL command right now :) ) and needs testing from “the other side” (terminal developers!).
But at the end of the day the two things do get confused in multiple terminal emulators. (And I knew you could set a terminal title with an escape sequence, but I didn’t know you could query it, and I didn’t know you could query the font. What apps even need these commands? Why do they exist?
I think a terminal should be more like HTML than JavaScript. It should just display colors and bold?)
Headless mode does fundamentally address this, because the idea is that you type into a different text box that than the one that displays command output !
Repeating the screenshot / explanation of how it works from awhile ago:
I have wondered if there is some way for bash/zsh to get headless mode. Then it would more of a no-brainer for terminals to implement.
But I think you need a totally separate input loop inside bash/zsh, using a Unix domain socket. That’s how it works in OSH, although it’s only like 50 lines or so. If someone thinks it can be done in bash or zsh, I would definitely pay a bounty out of our Github sponsors for that (like $1000 or more)
But yeah I definitely think the entire shell/terminal emulator interface needs to be rethought, and Oils has a 100% POSIX compatible shell, and bash-compatible shell, so it’s a good place to experiment. Feedback is very welcome!
ls and cat (coreutils), python3, ruby, OSH/YSH – these do proper escaping
Filenames are untrusted data, and should always be escaped when printing to a terminal. So this seems like another attack vector. Instead of RED you can do the thing of opening the calculator app and so forth:
printf '\e]0;iopen -a Calculator\a\e[21t\e]0;:accept-line\a\e[21t'
And yes it’s unfortunate that Unix is this soup/sludge of strings … which is why Oils is based on principled parsing, e.g. regular languages and grammars
Some kind of cookie is definitely a way to implement it, however it’s useful to think about the threat model first. The idea isn’t exactly like pledge() but close – the program running in the terminal is (more) trusted but the output isn’t (even when running over SSH). Therefore a cookie could safely be passed via an environment variable as malicious output is unlikely to be able to be affected by environment variables (and if it can there’s a different security problem).
That way reset can read the cookie (or more likely ncurses, I’m handwaving a lot as this would need ecosystem wide changes). This also solves the problem of tools with subcommands that behave differently – consider kubectl with get which prints output, but kubectl edit can run an editor, the editor would be able to read the environment variable and allow itself to write to the screen.
(I wonder if a heuristic that would allow prototyping this is to make the shell work out if what it is running is linked to ncurses or not, clearly an awful heuristic that won’t cope with static/Go/Rust programs but could be interesting to play with. I think the biggest challenge isn’t the escape sequence design but working out how to deploy this.)
This is likely the best that can be done unless we start talking about kernel changes, e.g. cap_rights_limit from Capsicum with added capabilities for writing escape sequences and a recvmsg-like system call that can tell you what capabilities the writer had…
I would be interested in co-designing / implementing such a scheme in a shell, if it improves security (https://www.oilshell.org/)
But I honestly don’t understand how these terminal escapes are even used. Why aren’t they always off? Repeating my question above:
And I knew you could set a terminal title with an escape sequence, but I didn’t know you could query it, and I didn’t know you could query the font. What apps even need these commands? Why do they exist?
I think a terminal should be more like HTML than JavaScript. It should just display colors and bold?
So is the idea that the shell has higher privileges than cat ? So each time it executes a child, it drops privileges
And then it restores privileges for the user’s own shell config to use?
Because some people have shell configs that query the window title? And query the font ???
And the blog post mentioned something about resizing the terminal to 32 or whatever. I am curious what breaks if you just disable all of them …
I’ve been using jj pervasively recently and I think my favorite part is the plain confidence that comes from switching between branches when files are always tracked. I can be editing one commit, realize something about the code that recontextualizes a completely separate branch, and then just stop, jj edit the other branch immediately, and come right back. The important part is that never once in this flow do I have to think about “what about my current changes?” because they’re always saved. Always.
This workflow astounded me two months ago, but now I do this branch switching daily. Never having to think about git stash ever again is so empowering.
It’s a bit ironic, I’ve tried jujutsu before (when it was probably a bit less stable) and somehow couldn’t wrap my head around it that well. Went back to git almost immediately.
I think your comment about not thinking about current changes just made it “click” a bit for me :)
I’ve long ago started telling other people how bad the git interface is and somehow I wanted jj to just be a better UI, when it is a (hopefully) better workflow - with a better UI.
I gonna have to give it another go down the road…
Never having to think about git stash ever again is so empowering.
That sounds incredible. One of the worst parts about git is managing worktrees, or stashes, or multiple repo copies entirely (for when things get really hairy or critical).
The Oils project always welcomes contributors, and has had 82 contributors over the years. It’s a new Unix shell, but also a vision for simpler computing, which I think is practical to adopt – because it doesn’t insist on rewriting the world. It uses what’s already there, and what’s known to work.
It’s all Python, C++, and shell - the most bog standard and proven/reliable technologies. Although we use those to implement domain-specific languages, i.e. the “middle-out” style:
If you want to read a lot of code, I think our code is very readable, once you understand the patterns. The syntax for the entire language is described in ~650 lines of algebraic data types:
It’s detailed, but conceptually, that’s all it really takes to understand the project. I often say it’s “Crafting Interpreters + 20 or so syscalls”. So anyone who understood 50% of Crafting Interpreters, and knows Python, should be able to contribute to Oils.
The other thing that is a “must” is a healthy respect for testing, and what I think of as “adversarial testing” – i.e. trying to prove yourself wrong, not prove yourself right.
Contributors may not end up writing large volumes of code. It’s more about reading, as you say! And thinking
We try to write as little code as possible – the whole point of the short style is to reduce the number of bugs. I think it’s pretty accepted that the number of bugs is proportional to source code size, regardless of language.
I’d also claim that it isn’t linearly proportional – my experience is that once code reaches a certain size, the number of bugs grows nonlinearly. Bugs lead to other bugs – including performance bugs, which often end up as permanent properties of the software.
I should also add that you don’t have to be paranoid about introducing bugs when contributing to Oils.
You can kinda just hack and see what works
We have so many tests that almost all breakages should be caught – and if they aren’t, that’s the fault of the tests, not the person making the change
So IMO it is kind of fun to work on a project like that – where you don’t have to be paranoid about what you can do, and can just try stuff. It enables much faster learning.
Also, being in Python means that your “compile” loop is non-existent. It is < 30 ms to iterate, which is practically instant.
But then we compile the Python to C++, which makes the code 2x-50x faster. We are in the realm of hand-written C++/Go/Rust, but we write typed Python.
So it’s a quirky toolchain that we spent a lot of effort on, but the result is the best of both worlds IMO.
The typed Python is also nice … I do aggressive global refactorings with types. But sometimes I ignore the types when I’m trying to debug/reverse engineer an algorithm, and then fix types later. So you can choose whatever is faster.
Honestly I never read any of the ASDL files. I just tried the language, asked for a few pointers for a feature I wanted to see, wrote some spec tests and started tinkering with the python code.
I believe this project has a very easy entry (e.g. implementing simple builtin methods/functions, or even something in the stdlib which is just shell itself)!
And then the curve goes up until you land at C++ transpilation, the handwritten GC, etc. - way above my head at least!
Really something for anyone if you ask me!
And Andy is super responsive & helpful in providing guidance and tips on zulipchat/github - super motivating.
It was also super rewarding to get something from the NLNet fund for some IMO very small contributions!
I switched to oils-for-unix a few weeks ago as main shell and am very sure it will become wildly popular, even though the syntax & features seem not as nice as the ones of modern alternatives like nushell/elvish/etc. The reasons are quite simple:
Opposite to the other shells it tries hard to be exterior first.
It upgrades bash instead of being completely new & from scratch. (the osh part should be POSIX compliant!). This will allow codebases with thousands of lines of shell like kubernetes or build systems to adapt it with almost zero changes and then gradualy improve the code & slowly move from the legacy osh part to ysh
The requirements are basically a GCC, libc, make and bash. This makes it actually usable for bootstrapping. IIRC most other modern shells lack a lot in this regards. I compiled Oils on Android and it’s extremely easy to port to anywhere else.
I’m going for the “shell is not supposed to do it” sacred cow in the name of productivity. Only semantically aware software has the chance to make you productive. Understand the output, interact with the objects on the screen, capture the semantics of the interaction so it could be replayed.
It’s interesting for me how everyone loves command line completion which is basically semantic understanding of inputs of programs but understanding outputs by the shell, the next logical step is always out of scope.
Just learned about your shell here. Oils has a headless mode which technically allows to build out some of your ideas. But I definitely follow iTerm2 creators’s quote. There is a LOT of legacy that needs to be overcome! I wish you best luck with it!
Go ahead! I would like these ideas to become commonplace so they wouldn’t be uphill battle and “the shell is not supposed to do that” will be a thing of the past.
That will not publish the file you have in front of you but the file at the build path of {packages.website}. Or do I miss something? :)
It obviously has to built the publish script everytime, which is weird - but I guess that’s kind of the joke of Nix.
I could imagine other solutions could work better; e.g. in the publish script first run nix build .#website and publish that output.
Wow, this is exactly the reaction I kinda imagined when I saw paperwm.
Before doing the switch from awesome to paperwm I found Niri and decided that I’ll move directly to wayland+Niri instead. Though I kinda haven’t had the guts to leave all my X helper scripts behind and actually move to wayland yet…
I guess this is what I needed to actually seriously consider switching to wayland soon. Thanks!
I found paperwm first, and it just really didn’t work for me, leading to me ignoring niri for a fairly significant amount of time by association. This post gave me the nudge to actually try it out, and I’m very pleasantly surprised! Going to be switching to it fully for a while to really get a feel for it, and then decide if I’ll stay definitively.
Curious - what do you like about niri that PaperWM didn’t have – or PaperWM has that niri doesn’t, I guess.
Running smoothly! (+ decoration handling, scaling, shortcut management…) Using PaperWM felt like replacing my touchpad with a cheese grater comparatively (I’m typically a labwc user).
I’ve used both, and there are trade-offs.
With PaperWM, you get all the integration that Gnome provides, mainly in terms of managing hardware devices. You can provide all of this for Niri, and I have (originally for Sway, but it was trivial to bring my Sway configuration over to niri), but there’s no turnkey solution.
On the other hand, niri is much faster and smoother than PaperWM, and less buggy (because PaperWM is forced to keep up with Gnome Shell compatibility and interaction with other extensions). I also find that it handles the interaction between workspaces and multiple displays better than PaperWM.
Generally, I haven’t had many complaints with PaperWM (knocks on wood) or found it to be buggy. I do worry about the keeping up with GNOME – right now development with PaperWM seems active enough, but I can easily see a scenario where the developers move on and it stops working with newer versions of GNOME. If that day comes, I guess I’ll definitely be moving over to niri whether I want to or not.
Simplex sounds interesting. Giving up on having any kind of identifier that a human can read remember and type makes for interesting UX challenges but if this can be handled by address book management it may provide a much better way of exchanging encryption keys.
I am interested in review of the crypto and privacy properties. There doesn’t seem to be anything on the IACR eprint server. Has nobody looked at this yet?
There was a review by trail of bytes in 2024, if that’s what you mean: https://simplex.chat/blog/20241014-simplex-network-v6-1-security-review-better-calls-user-experience.html
Good. GOOD.
I’m seriously considering trying Oils, but purely for the footgun-elimination aspect. Having a feature-complete, stable, default shell is most definitely a feature as far as I’m concerned.
Very underappreciated argument, thanks for bringing it up! I think it’s a question of how much convenience (features) you’d want on a modern shell. IMO the “feature-complete” POSIX Standard is not enough. And the Bash features make everything only confusing/are not really “complete”.
If only GitLab introduced seamless Pijul support! That is, make sure that both Pijul and Git clients can interact with repos in a sensible way. Then I could learn the ins and outs of Pijul with an easy fallback in case of catastrophic usability failure.
(I just learned that Nest (the Pijul hosting solution) is not yet source-available, which means self-hosting is not feasible for the foreseeable. 😢)
Yeah I guess as long as there is no open hosting solution the gate is kinda closed for wide adoption :/ tbh I haven’t looked into it’s ecosystem for a long time now. I wonder how pijul vs jujutsu would compare/if a pijul “backend” could be supported by jujutsu.
Does Radicle do any kind of CI? I couldn’t find anything about that in their FAQ.
Uh good point. I know there was some dicussion in the zulip when I looked into it, but it seems like there’s currently only a poc.
SimpleX seems cool but:
Surely this is how almost every chat app is getting pwned though?
Not sure. The question is who is the attacker. If it’s state agency, then definitely not. If a state agency is targeting you, you probably don’t stand a chance. But given how state leaders are shifting more and more to the fascist extremes (left and right…), complete citizen observation isn’t just a topic for china anymore. So not wanting any data on the server is IMO a very valid concern, because that makes “observe everyone” much harder.
I followed the same train of thought and decided to go with Jujutsu (jj). Like pijul, it also has new, strong design ideas – but it is less radical. Unlike pijul, it is compatible with git repositories and forges, which means that it has a much easier path to adoption. (My application is “much better experience than git when rewriting histories in preparation for one or several github PRs”.)
Yes true! I have it in my backlog to take another look (didn’t get it last time i tried it, I didn’t realize I should change the workflow :)). I wonder if the longterm goal of Jujutsu is to introduce an alternative backend which is more “native” to Jujutsu. In that case it would be taking “the long road” very similarly to Oils. Reimplement, then improve.
I have the feeling this is a currently very liked approach. E.g. coreutils is one other example (thought this is more about the language/development ergonomics than features/usage economics).
jj already has it’s own background but from the beginning it has had git compatible storage as well as sync protocol.
I also bounced of half a year ago or so, but things have come a long way now and it’s really much better now and usable as a daily driver.
Yes, see our roadmap.
Very neat! I recently realized that shells strength is pipelines/streaming of external tools, but it’s weak at adding a function in the middle of such a stream: https://blog.yosemitesam.ch/shell-a-stream-processing-language/ (refresh on error 500)
I think your post kinda boils down to a similar observation; Shell pipelining is also weak at user interactivity.
I want to say thanks to @dgl for reporting this, and this article is also expertly written.
I also have to say for me personally its quite embarrassing because as I told @dgl when he reported this: I’ve studied his work before and made it a note to test Ghostty against his past discoveries prior to release. And I… quite simply forgot. I didn’t make an issue for myself so it slipped away and here I am with egg on my face. I’m sorry!
But, I appreciate @dgl for the security report, reviewing the fix, and continuing to be an active user of the terminal. I also shared with him some broader thoughts on terminal security in general. Addressing terminal security in a more fundamental way is one of the first proposals I want to make regarding terminal innovation.
My thinking is still too early and under-researched for a formal proposal. But my general feeling having built a terminal over the past 2 years is that the security surrounding escape sequences is fundamentally flawed and poking one by one at these sequences to try to make them individually safe doesn’t feel like the right long term solution.
The surface area is too large and the complexity of some of the newer sequences too high (i.e. Kitty Graphics) to be confident in secure implementations. DoS is far too easy with terminals (Ghostty has a handful of known DoS attacks, but so does pretty much every other terminal I know of). And some legacy sequences are just kind of shocking to have immediately available. For example, DECCOLM is available in macOS Terminal.app. If you issue a DECCOLM (
CSI ? 3 hI believe), Terminal.app will physically resize the window and lock it to 132 columns. You can very easily crash Terminal.app at anytime by sending enough of these (a DoS attack). There are many more.Part of my thinking is trying to design a mechanism that can effectively create something akin to either CPU protection rings or OpenBSD’s pledge() syscall. Whatever the mechanism, the general idea is: reduction of capability.
For example, it makes sense for a shell to be extremely powerful. It’s literally a code execution device. However, it doesn’t make much sense for
catto be able to execute arbitrary escape sequences (which it does today in any terminal). Whether it is the shell orcatitself, something should be able to tell a terminal: “hey, only execute some subset of escape sequences (or none at all).” For cat, that might be none or perhaps just SGR sequences (styles, colors, etc.). The insecurity of cat-ing or tail-ing any form of data that could contain user-generated data is hopefully well known at this point…As I said before, this thinking is all still very raw and I don’t have anything concrete to proposal. I want to share this because I want folks to know that I’m thinking about it, and perhaps others may be interested in thinking about it with me… if so, please contact me. I’ve already shared this line of thinking with @dgl as well.
I wanted to send you a mail for some time but here we go :) @andyc, author of Oils has designed the currently very simple FANOS Protocol. It itself has nothing to do with escape sequences. It’s merely for seperating inputs & outputs of commands. Ie the terminal sends the shell
EVAL cat blaand 3 file descriptors, the shell executescat blaand uses the fds for STDIN/STDOUT/STDERR. The terminal has the other ends of the fds open and talks to/reads fromcatdirectly.Why is this relevant? For your example there’s currently no way for the terminal to know who sent what output (e.g. if I run something like
{sleep 0.1; printf 'EVIL ESCAPE';}& cat huge-file). FANOS would pass the control for merging the output streams over to the terminal. This solves many issues where currently workarounds need to be used, like overly complicatedPROMPT_COMMAND, etc. And it could be a cornerstone to also improve security, interactive features, etc. as the terminal suddenly has much more knowledge. IMO this is the kind of protocol which should’ve been developed 30 years ago.If you’re interested, we’d be super glad if you would join the #shell-gui channel on the Oils Zulip chat for feedback & discussions. The protocol is also super raw (only has
EVALcommand right now :) ) and needs testing from “the other side” (terminal developers!).Thanks for mentioning the Oils headless mode! So the fundamental issue is that 2 things are being confused:
I’m actually having trouble following this exploit, e.g. how the
^Gin the font query reply gets confused with the^Ginset -o vimode in zsh :https://www.openwall.com/lists/oss-security/2022/11/10/1
But at the end of the day the two things do get confused in multiple terminal emulators. (And I knew you could set a terminal title with an escape sequence, but I didn’t know you could query it, and I didn’t know you could query the font. What apps even need these commands? Why do they exist?
I think a terminal should be more like HTML than JavaScript. It should just display colors and bold?)
Headless mode does fundamentally address this, because the idea is that you type into a different text box that than the one that displays command output !
Repeating the screenshot / explanation of how it works from awhile ago:
https://www.oilshell.org/blog/2023/12/screencasts.html#headless-protocol-oils-web_shell
https://lobste.rs/s/ineb98/ghostty_1_0_is_coming#c_fb1rw7
I have wondered if there is some way for bash/zsh to get headless mode. Then it would more of a no-brainer for terminals to implement.
But I think you need a totally separate input loop inside bash/zsh, using a Unix domain socket. That’s how it works in OSH, although it’s only like 50 lines or so. If someone thinks it can be done in bash or zsh, I would definitely pay a bounty out of our Github sponsors for that (like $1000 or more)
But yeah I definitely think the entire shell/terminal emulator interface needs to be rethought, and Oils has a 100% POSIX compatible shell, and bash-compatible shell, so it’s a good place to experiment. Feedback is very welcome!
A related issue I didn’t notice until working on a shell is that tons of programs print unescaped filenames to the terminal! Consider:
Now try
$prog "$red"for any program that opens a file. What error message does it print?On my Debian machine, you will actually see RED for:
But you won’t for:
Filenames are untrusted data, and should always be escaped when printing to a terminal. So this seems like another attack vector. Instead of RED you can do the thing of opening the calculator app and so forth:
And yes it’s unfortunate that Unix is this soup/sludge of strings … which is why Oils is based on principled parsing, e.g. regular languages and grammars
Something like two new terminal codes:
allow_only(cookie, codes…)tells the emulator to only respond to the following control codes until…relax(cookie)will re-enable the previously constrained codes.It’s up to the sender / shell to generate a secure cookie. Of course, a malign
catcould wedge your terminal beyondreset’s ability to fix it. Hmm…Some kind of cookie is definitely a way to implement it, however it’s useful to think about the threat model first. The idea isn’t exactly like
pledge()but close – the program running in the terminal is (more) trusted but the output isn’t (even when running over SSH). Therefore a cookie could safely be passed via an environment variable as malicious output is unlikely to be able to be affected by environment variables (and if it can there’s a different security problem).That way
resetcan read the cookie (or more likely ncurses, I’m handwaving a lot as this would need ecosystem wide changes). This also solves the problem of tools with subcommands that behave differently – considerkubectlwithgetwhich prints output, butkubectl editcan run an editor, the editor would be able to read the environment variable and allow itself to write to the screen.(I wonder if a heuristic that would allow prototyping this is to make the shell work out if what it is running is linked to ncurses or not, clearly an awful heuristic that won’t cope with static/Go/Rust programs but could be interesting to play with. I think the biggest challenge isn’t the escape sequence design but working out how to deploy this.)
This is likely the best that can be done unless we start talking about kernel changes, e.g. cap_rights_limit from Capsicum with added capabilities for writing escape sequences and a recvmsg-like system call that can tell you what capabilities the writer had…
I meant that the cookie would be created by the shell and consumed by the terminal. No need for an environment variable!
I would be interested in co-designing / implementing such a scheme in a shell, if it improves security (https://www.oilshell.org/)
But I honestly don’t understand how these terminal escapes are even used. Why aren’t they always off? Repeating my question above:
So is the idea that the shell has higher privileges than
cat? So each time it executes a child, it drops privilegesAnd then it restores privileges for the user’s own shell config to use?
Because some people have shell configs that query the window title? And query the font ???
And the blog post mentioned something about resizing the terminal to 32 or whatever. I am curious what breaks if you just disable all of them …
Thank you Mitchell, your handling of this has been exemplary.
I’ve been using jj pervasively recently and I think my favorite part is the plain confidence that comes from switching between branches when files are always tracked. I can be editing one commit, realize something about the code that recontextualizes a completely separate branch, and then just stop,
jj editthe other branch immediately, and come right back. The important part is that never once in this flow do I have to think about “what about my current changes?” because they’re always saved. Always.This workflow astounded me two months ago, but now I do this branch switching daily. Never having to think about
git stashever again is so empowering.(Posted to the wrong subject first lol)
It’s a bit ironic, I’ve tried jujutsu before (when it was probably a bit less stable) and somehow couldn’t wrap my head around it that well. Went back to git almost immediately. I think your comment about not thinking about current changes just made it “click” a bit for me :) I’ve long ago started telling other people how bad the git interface is and somehow I wanted jj to just be a better UI, when it is a (hopefully) better workflow - with a better UI. I gonna have to give it another go down the road…
That sounds incredible. One of the worst parts about git is managing worktrees, or stashes, or multiple repo copies entirely (for when things get really hairy or critical).
The Oils project always welcomes contributors, and has had 82 contributors over the years. It’s a new Unix shell, but also a vision for simpler computing, which I think is practical to adopt – because it doesn’t insist on rewriting the world. It uses what’s already there, and what’s known to work.
It’s not for everyone, and I specifically list 6 reasons why here, so as not to waste anyone’s valuable time: https://github.com/oils-for-unix/oils/wiki/Where-Contributors-Have-Problems
It’s all Python, C++, and shell - the most bog standard and proven/reliable technologies. Although we use those to implement domain-specific languages, i.e. the “middle-out” style:
This makes the code very short:
If you want to read a lot of code, I think our code is very readable, once you understand the patterns. The syntax for the entire language is described in ~650 lines of algebraic data types:
https://github.com/oils-for-unix/oils/blob/master/frontend/syntax.asdl
And the runtime is even smaller - https://github.com/oils-for-unix/oils/blob/master/core/runtime.asdl
It’s detailed, but conceptually, that’s all it really takes to understand the project. I often say it’s “Crafting Interpreters + 20 or so syscalls”. So anyone who understood 50% of Crafting Interpreters, and knows Python, should be able to contribute to Oils.
The other thing that is a “must” is a healthy respect for testing, and what I think of as “adversarial testing” – i.e. trying to prove yourself wrong, not prove yourself right.
Contributors may not end up writing large volumes of code. It’s more about reading, as you say! And thinking
We try to write as little code as possible – the whole point of the short style is to reduce the number of bugs. I think it’s pretty accepted that the number of bugs is proportional to source code size, regardless of language.
I’d also claim that it isn’t linearly proportional – my experience is that once code reaches a certain size, the number of bugs grows nonlinearly. Bugs lead to other bugs – including performance bugs, which often end up as permanent properties of the software.
There’s also a separate shell GUI project I mentioned here, if anyone wants to run with it, and write a ton of code: https://lobste.rs/s/ineb98/ghostty_1_0_is_coming#c_fb1rw7
I should also add that you don’t have to be paranoid about introducing bugs when contributing to Oils.
You can kinda just hack and see what works
We have so many tests that almost all breakages should be caught – and if they aren’t, that’s the fault of the tests, not the person making the change
So IMO it is kind of fun to work on a project like that – where you don’t have to be paranoid about what you can do, and can just try stuff. It enables much faster learning.
Also, being in Python means that your “compile” loop is non-existent. It is < 30 ms to iterate, which is practically instant.
But then we compile the Python to C++, which makes the code 2x-50x faster. We are in the realm of hand-written C++/Go/Rust, but we write typed Python.
So it’s a quirky toolchain that we spent a lot of effort on, but the result is the best of both worlds IMO.
The typed Python is also nice … I do aggressive global refactorings with types. But sometimes I ignore the types when I’m trying to debug/reverse engineer an algorithm, and then fix types later. So you can choose whatever is faster.
Honestly I never read any of the ASDL files. I just tried the language, asked for a few pointers for a feature I wanted to see, wrote some spec tests and started tinkering with the python code.
I believe this project has a very easy entry (e.g. implementing simple builtin methods/functions, or even something in the stdlib which is just shell itself)! And then the curve goes up until you land at C++ transpilation, the handwritten GC, etc. - way above my head at least!
Really something for anyone if you ask me! And Andy is super responsive & helpful in providing guidance and tips on zulipchat/github - super motivating.
It was also super rewarding to get something from the NLNet fund for some IMO very small contributions!
I switched to oils-for-unix a few weeks ago as main shell and am very sure it will become wildly popular, even though the syntax & features seem not as nice as the ones of modern alternatives like nushell/elvish/etc. The reasons are quite simple:
oshpart should be POSIX compliant!). This will allow codebases with thousands of lines of shell like kubernetes or build systems to adapt it with almost zero changes and then gradualy improve the code & slowly move from the legacyoshpart toyshNext Generation Shell because I wrote it. Hi, xiaq!
For scripting for now but UI is in progress.
Why? Fully fledged programming language that adds domain specific parts: syntax and facilities.
While the programming language is way more aligned with how I think than other shells, the biggest differentiator would be the UI.
https://github.com/ngs-lang/ngs/wiki/UI-Design
I’m going for the “shell is not supposed to do it” sacred cow in the name of productivity. Only semantically aware software has the chance to make you productive. Understand the output, interact with the objects on the screen, capture the semantics of the interaction so it could be replayed.
It’s interesting for me how everyone loves command line completion which is basically semantic understanding of inputs of programs but understanding outputs by the shell, the next logical step is always out of scope.
Have a nice week!
Just learned about your shell here. Oils has a headless mode which technically allows to build out some of your ideas. But I definitely follow iTerm2 creators’s quote. There is a LOT of legacy that needs to be overcome! I wish you best luck with it!
The plan for NGS from early on was to have two pluggable front ends: web and terminal.
Thanks!
Hiya! The UI design has some super awesome ideas, looking forward to trying it out and
stealinglearning from it :)Go ahead! I would like these ideas to become commonplace so they wouldn’t be uphill battle and “the shell is not supposed to do that” will be a thing of the past.
The code is probably not gonna work (
rec {required and maybe more) but wouldn’t something like this work?:That will not publish the file you have in front of you but the file at the build path of
{packages.website}. Or do I miss something? :) It obviously has to built the publish script everytime, which is weird - but I guess that’s kind of the joke of Nix.I could imagine other solutions could work better; e.g. in the publish script first run
nix build .#websiteand publish that output.No, you’re on point, this would publish the purely built version of the website.
This would work too, but in the end it would have the same effect!