I only skimmed the article, but the first animation, while looking nice, doesn’t seem correct to me.
The new process (ls, in this example) is described as sending its output to the shell, but in reality the forked process sends its output directly to the inherited TTY. The shell never sees any of it, which is also the reason it can’t do anything about background jobs messing up the output of the foreground job.
It is oversimplifying, which is fair given the introductory nature and all that; it is whatever is mapped to the file descriptor slots for STDIN/STDOUT/STDERR in the new child. That can be be some PTY, other files, or “in the olden days” even empty (not as in /dev/null, as in not allocated).
It was a fun trick to mess with SUID binaries in this way, as the spare allocation requirement means that if you say, close(2); exec(); the next open() will also become STDERR, so find a suid binary where you had partial control of fprintf(stderr, “%s went wrong”, “exploit goes here”) there were some privilege escalation opportunities. For that reason, kernel or libc these days tend to make sure /dev/null go there unless set. Even non-maliciously it cause some terrible bugs (whatever you logged also corrupting the file you were working with etc.).
The sparse allocation requirement for open() absolutely sucks.
So in reality it can be both, some or neither. It is a chain of trust thing - the shell can run new jobs in whichever way it seems fit (close/dup/open other things for stdin/stdout/stderr), including nesting new terminal emulators (tmux, screen and so on).
This is really a big pain-point: the shell isn’t certain of what it is running and feeding it instructions. The terminal emulator isn’t certain of what the shell is running. The thing that is running isn’t certain of what it is supposed to be, interactive or in a pipeline – and have few options for being both. It can only guess (isatty() style shenanigans testing various “benign” pty- dependent ioctls and checking for failure).
Now with this premise, try and get buffering, synchronization and signal propagation “right”…
The user types in text which is buffered in the PTY’s STDIN line buffer. The user presses and the pty’s STDIN line buffer is sent to the shell
How would tab completion work? I suspect some really old terminals and shells would buffer the entire line, but modern tooling likely works on a per-character basis. Another example is Fish Shell, which has native type-ahead history/search that appears as you type (I think there are extension that can do this in Zsh too?)
The article discusses non-canonical input processing (aka “raw mode”) and how it differs from the default processing that occurs for a tty where the line discipline (usually in the kernel) provides extremely basic line editing capabilities. Indeed, for better control over the display and for advanced features like tab completion, a modern shell will have the tty in raw mode all the time and will switch it back to canonical mode when starting a process at the request of the user.
There’s a lot of great info in here, but I feel Rust isn’t a great language for this kind of thing, there’s a lot of rusty boilerplate clogging up the parts that are actually about how terminals work :( I know enough rust to read it anyway, but it’s definitely something that’s in the way, and hampers my understanding of the important concepts.
I think it just depends on what you’re used to. Given how much of this stuff is low-level system and library calls, it makes sense to pick something that has good support for FFI and reasonable wrappers around those facilities, both of which are true of Rust.
I think it’s hard to write something like this, where the goal seems to be to inform and teach, without it becoming copy-paste production code in a variety of systems. In that sense it seems preferable to write the examples in a way that they can be used directly for new software, which is unlikely to be in C in 2021.
This is really excellent; thanks for sharing! Love to see more of these introductory yet still deeply-technical overviews. The interactive animation widget thing in particular is a great touch.
Really great article! I can’t judge the quality of the explanations (though they seemed pretty accurate to me), but the animations really bring something to it. I can only wish we had more animations like that in other articles :)
I only skimmed the article, but the first animation, while looking nice, doesn’t seem correct to me.
The new process (
ls, in this example) is described as sending its output to the shell, but in reality the forked process sends its output directly to the inherited TTY. The shell never sees any of it, which is also the reason it can’t do anything about background jobs messing up the output of the foreground job.It is oversimplifying, which is fair given the introductory nature and all that; it is whatever is mapped to the file descriptor slots for STDIN/STDOUT/STDERR in the new child. That can be be some PTY, other files, or “in the olden days” even empty (not as in /dev/null, as in not allocated).
It was a fun trick to mess with SUID binaries in this way, as the spare allocation requirement means that if you say, close(2); exec(); the next open() will also become STDERR, so find a suid binary where you had partial control of fprintf(stderr, “%s went wrong”, “exploit goes here”) there were some privilege escalation opportunities. For that reason, kernel or libc these days tend to make sure /dev/null go there unless set. Even non-maliciously it cause some terrible bugs (whatever you logged also corrupting the file you were working with etc.).
So in reality it can be both, some or neither. It is a chain of trust thing - the shell can run new jobs in whichever way it seems fit (close/dup/open other things for stdin/stdout/stderr), including nesting new terminal emulators (tmux, screen and so on).
This is really a big pain-point: the shell isn’t certain of what it is running and feeding it instructions. The terminal emulator isn’t certain of what the shell is running. The thing that is running isn’t certain of what it is supposed to be, interactive or in a pipeline – and have few options for being both. It can only guess (isatty() style shenanigans testing various “benign” pty- dependent ioctls and checking for failure).
Now with this premise, try and get buffering, synchronization and signal propagation “right”…
Yea I also question this part:
How would tab completion work? I suspect some really old terminals and shells would buffer the entire line, but modern tooling likely works on a per-character basis. Another example is Fish Shell, which has native type-ahead history/search that appears as you type (I think there are extension that can do this in Zsh too?)
The article discusses non-canonical input processing (aka “raw mode”) and how it differs from the default processing that occurs for a tty where the line discipline (usually in the kernel) provides extremely basic line editing capabilities. Indeed, for better control over the display and for advanced features like tab completion, a modern shell will have the tty in raw mode all the time and will switch it back to canonical mode when starting a process at the request of the user.
There’s a lot of great info in here, but I feel Rust isn’t a great language for this kind of thing, there’s a lot of rusty boilerplate clogging up the parts that are actually about how terminals work :( I know enough rust to read it anyway, but it’s definitely something that’s in the way, and hampers my understanding of the important concepts.
I think it just depends on what you’re used to. Given how much of this stuff is low-level system and library calls, it makes sense to pick something that has good support for FFI and reasonable wrappers around those facilities, both of which are true of Rust.
I would have preferred C, so there are no wrappers :) Not necessarily for production code but for pedagogy.
I think it’s hard to write something like this, where the goal seems to be to inform and teach, without it becoming copy-paste production code in a variety of systems. In that sense it seems preferable to write the examples in a way that they can be used directly for new software, which is unlikely to be in C in 2021.
This is a good point.
This is really excellent; thanks for sharing! Love to see more of these introductory yet still deeply-technical overviews. The interactive animation widget thing in particular is a great touch.
Really great article! I can’t judge the quality of the explanations (though they seemed pretty accurate to me), but the animations really bring something to it. I can only wish we had more animations like that in other articles :)