One thing that I think is interesting about plan 9 is that, in the later papers, they present a complicated song-and-dance of files you need to read from and write to in order to perform an operation–and then say that you probably don’t want to do that since it’s too complicated, and then give you a library to link to, so you can just call C functions instead of manipulating files.
Another limitation of the “everything is a file approach” is that it’s not necessarily evident how to use a subsystem. While it’s true that any program can just open /my-subsystem/control, and start writing commands into it–the fact that /my-subsystem/control looks like a file isn’t so useful, if you don’t know what commands to write into it!
UNIX and Plan 9 have drawn favorable comparisons to Smalltalk. In many ways, Plan 9 feels like UNIX, but even more Smalltalk-y. The main difference is that the core operation in Smalltalk is sending a message, and messages can have customized names and meanings, which powers tools like the Smalltalk browser, which let you explore what messages an object will respond to. Plan 9 only exposes pre-defined messages like “open file,” “write to file”, etc.
One really cool thing that Plan 9 does, which I think is useful for both programming systems and operating systems, is that namespacing removed “ambient capability.” There’s no global “connect to the network” thing, for example. If you can connect to the network, it’s because the parent of your process has given you a file(ish), and you connect to the network by writing to that. Importantly, you have no clue if this is the “actual” network, or some sort of firewall that your parent process has set up. This is a powerful idea that underlies both Smalltalk and Plan 9–what matters is the messages that you send and the protocols that they respond to.
One modern-day system that’s working on this is Fuchsia. It has Plan 9-like namespacing, but a more Smalltalk-like messaging system. For example, here’s what the messages look like for a DNS lookup in Fuchsia. Your process can do a DNS lookup, if it’s given something that conforms to the Lookup protocol, but that could be the OS’s root DNS resolver, or something else.
I’m not sure what you’re referring to – perhaps the internals of dial(), but the benefits of a uniform file-based interface are there regardless of how it’s used. The uniformity around both system-visible operations and name resolution is the point.
The key point is that it’s trivial to write interfaces, wrappers, and interposition layers that are completely agnostic to the RPCs being made. The network stack (/net) can be exported and imported with completely generic tools. The mouse, keyboard, and display too – this is used for remote logins, giving a VNC-like experience from a 50 line shell script. The script itself doesn’t know anything about the device’s meta-protocol.
You don’t even need to know what interfaces exist in order to transparently and fully proxy, inspect, or re-route them.
Although programs can access factotum’s services through its file system interface, it is more common to use a C library that packages the interaction.
The section on protocol devices described the details of making and receiving connections across a network. The dance is straightforward but tedious. Library routines are provided to relieve the programmer of the details.
The window system also suggests using a library instead of interacting directly with the files.
You don’t even need to know what interfaces exist in order to transparently and fully proxy, inspect, or re-route them.
I agree that it’s easy to write servers which only proxy messages in the Plan9 system. I don’t know how that works in Fuchsia, but that’s really easy in Smalltalk. You can make an object which responds to all messages! You can then use whatever logic you want to ferry that message across the network, and then send that message to another object on another computer. I think the proxy server would actually look quite similar to the Plan9 version.
The Plan9 approach is much harder IMHO, if you want you actually manipulate the command stream. Consider for example, the /net/dns protocoll:
A client writes a request of the form domain-name type, where type is a domain name service resource record type. DNS performs a recursive query through the Internet domain name system producing one line per resource record found.
Let’s say I want all domains that exist to resolve to localhost, for some reason. In order to do this, the first thing I need to do is write a parser for this protocol. Not the hardest thing to do, but it still feels silly that this is a requirement.
The bigger issue is that this protocol is flawed. TXT DNS records (at least) can contain newlines. My simple parser will need to be modified to handle this edge case (as well as, presumably, every other DNS server, judging from the description of this DNS protocol).
Rather than pushing the parsing onto the user, leading to issues like this, in systems like Erlang, Smalltalk, and Fuchsia, the fact that messages are structured means, I think, that they’re much easier to work hit the ground running with.
The protocol is free form in theory, but it’s pretty much standardized in practice, at least for textual messages. The parsing is handled by the tokenize() library call in userspace, or by the parsecmd call in the kernel. The quoting is handled by the ‘%q’ format specifier. This handles edge cases pretty well.
In practice, I often need less code and effort to deal with this than to write code against most other RPC libraries. It just isn’t a pain point.
they present a complicated song-and-dance of files you need to read from and write to in order to perform an operation–and then say that you probably don’t want to do that since it’s too complicated, and then give you a library to link to
A similar thing happens in Linux (raw system interfaces are too complicated to use, or are undocumented, so you use a C library interface instead that provides a higher level abstraction). A mild case of this is the new io_uring system call, which is cumbersome to use, and periodically extended, so the author of the system call also maintains a C library liburing. A more extreme case is the raw system interface to the GPU, which AFAIK is undocumented, specific to a particular GPU vendor, and unstable (subject to change). Your only real choice is to use a library interface like Vulkan or OpenGL.
Does Fuchsia also work this way? The DNS lookup interface you linked to is provided by a library called “fuchsia.net”. How much work is that library doing in userspace, and what does the raw system call interface look like?
Does Fuchsia also work this way? The DNS lookup interface you linked to is provided by a library called “fuchsia.net”. How much work is that library doing in userspace, and what does the raw system call interface look like?
Fuchsia is designed around message passing and capabilities. The actual system calls are basically just there to let processes communicate with each other. The DNS lookup interface that I linked to is the lowest-level DNS interface. Below that, it’s implemented by talking to the kernel to set up the communication between the client process and the DNS server process (which implements the DNS lookup protocol).
namespacing removed “ambient capability.” There’s no global “connect to the network” thing, for example. If you can connect to the network, it’s because the parent of your process has given you a file(ish)
and this
they present a complicated song-and-dance of files you need to read from and write to in order to perform an operation–and then say that you probably don’t want to do that since it’s too complicated, and then give you a library to link to
are actually two sides of the same coin.
The simple, file-like namespaces let API developers fully separate privilege control.
The simple, file-like namespaces let API developers fully separate privilege control.
The E programming language is an example of a system which gives users complete privilege control, but without 9P’s hard-coded requirement that the only allowable verbs are file system operations. I think whether you have an extensible set of verbs is a distinct issue from whether your system has ambient capabilities.
This is a general theme I’ve noticed with Linux: All the stuff I want is basically there, it’s just way more complicated and annoying than I’d like it to be.
Another example: OpenBSD gets praised a lot (rightfully imo) for its “pledge” system. Linux has something to do the same job, seccomp-bpf, but it’s so hard to actually set up that its usage is mostly limited to a very small set of core services.
There is most of a decent capability-based OS buried somewhere in Linux, you can send around file descriptors for almost anything (processes, sockets, memory, even namespaces themselves!) and you have a lot of tools to eliminate ambient authority (which is of course still there by default).
edit: and of course Linux also has a subsystem literally named “capabilities” that has nothing to do with any of this and is just generally awful.
It’s close, but it doesn’t quite do what pledge does. For example, you can’t pledge the shell to avoid doing any network I/O, while allowing programs spawned by the shell to still work.
It feels right that permissions obey a hierarchy, but it turns out to be very useful to break that hierarchy. It’d be even better to do it deeper than just a parent/child relationship, but that gets really complicated, so that one layer is a reasonable tradeoff in practice.
What’s the point of not letting the shell directly access the internet, while still letting it spawn other processes that can access the internet? Or am I misunderstanding?
Imagine that you’ve managed to find some bug in your shell; you want to pivot it to a complete expoit, but you don’t have any syscalls that you can use. If you’re careful about locking things down, you may have unveiled only a few binaries in the path, so you can’t even run arbitrary programs. Now, not only do you have to find an exploit in the shell, you have to pivot it to execute a program that will do what you want, and maybe garble its arguments in a way that it will give you more access.
In a privsep daemon, you may also have a spawner process that primarily has the job of creating processes that do the work; that process needs very little permissions to work, but the workers may need to do more.
It’s not a panacea, but it makes your attacker’s life a lot harder, and makes it possible to ratchet down permissions far more tightly.
It’s worth noting that Unix is fundamentally incompatible with free use of per-process namespaces; you can’t safely have setuid in an environment where the user can arbitrarily change the filesystem environment that a setuid program is running in. Plan 9 deals with this problem by doing away with setuid, an option that’s not available to Linux.
Please, stop praising Plan 9. Plan 9 is too simple. Plan 9’s authors created very elegant and simple set of abstractions, but when these abstractions prevent them from solving some problem, they simply refuse to solve that problem!!!! For example, Plan 9 doesn’t have proper “mv” between different directories (I wrote on this previously: https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAPnZJGBejeA+9APimVY6+HH%3DfWZJfW%3DQXJaOEK-JhhESRP+cSg%40mail.gmail.com%3E ). As well as I understand, proper “mv” is incompatible with “elegant” Plan 9’s abstractions, so the authors simply decided to throw it away (??????).
Also, Plan 9 is written in special dialect of C, which is incompatible with proper C. Why the authors created special language? If they really decided to create new language, why they made it mostly similar to C? Why not to create some really innovative language, such as Rust? So, instead of choosing between “normal compatible C” and “real innovation”, they chose worse sides of both. Look at any of Plan 9 sources, for example: http://git.9front.org/plan9front/plan9front/30c5296f32b87d83529d772732726891e1261c9c/sys/src/cmd/mv.c/f.html . What you see is absolutely obsolete programming practices. Raw C string manipulation. Raw pointers. And all this in userspace. I understand why C is used in kernels. But user space?
Plan 9 is based on writing text commands in special text files. This is absolutely untyped protocol. It ignores all innovation based on type systems.
Compare this with true revolutionary Fuchsia OS. Fuchsia authors care about performance. “Simple and slow” is not their method. (Very small) kernel is written in C (the authors had reasons to do so), but most of userspace is written in Rust. Communication is based on typed Fuchsia IPC protocol
Also, Plan 9 is written in special dialect of C, which is incompatible with proper C. Why the authors created special language? If they really decided to create new language, why they made it mostly similar to C?
Because when they initially made C, they realized they got things wrong, and changed them.
Plan 9’s abstractions make writing perfomant software very difficult (if possible), as noted here:
And yet, it tends to perform better on many basic system tasks. Obviously not everything – Linux has had a lot of optimization work put into it, and having a monolithic kernel and VDSOs instead of userspace file servers for some services cases gives a pretty big advantage to Linux – but Plan 9 keeps up surprisingly well;
Small examples: pipes are faster on 9front than on Linux, as is process spawning or task switching. Measure it yourself and see – in practice, performance tends to be quite acceptable. The last time I checked, on comparable hardware (recent Ryzen), execing a binary was about 10x faster on Plan 9 than on Unix.
It’s not because of any real effort in this space, Simplifying things tends to lead to less work being done, which leads to reasonable performance.
Plan 9’s abstractions make writing perfomant software very difficult (if possible)
Having used Plan 9 recently, one of its most notable characteristics is its blistering performance - it’s far more responsive than any mainstream desktop or mobile OS I’ve used in recent years, even when running on ancient hardware or in emulation.
I will agree that it makes writing efficient software difficult or impossible. But, the level of simplicity means that it’s still orders of magnitude more responsive than any Web or Electron app, and usually faster than a native app in a modern OS.
…in a February 2000 slideshow, Pike noted: “…although Alef was a fruitful language, it proved too difficult to maintain a variant language across multiple architectures, so we took what we learned from it and built the thread library for C.”
Basically, programming language tech in the early/mid 90’s wasn’t up to the task of making something Better Enough To Be Worth It, at least as far as the Plan 9 team knew.
Basically, programming language tech in the early/mid 90’s wasn’t up to the task of making something Better Enough To Be Worth It, at least as far as the Plan 9 team knew.
Hmmm. I’m not sure that’s true; I think that the real issue was that the Plan 9 team fundamentally disagreed with the philosophy of the things that were in my opinion Better Enough To Be Worth It - Lisp and Smalltalk.
They didn’t see those languages, and their approach to systems, as even being Better, let alone Worth It.
This may also have been related (though I don’t know which way around the relationship points) to Bell Labs’ plan to target embedded development.
Addition: Plan 9’s C dialect ignores “const” specifier!!!!! This means that the authors managed to make the language, which is even more unsafe than C!!!! (How this was possible? :) ). They throw away C’s const-correctness…
(But yes, okay, it seems I have old information. I get this info from original Thompson’s description of that C dialect. It seems the authors added this warning since that time.)
That’s probably the right call anyway, const in C is next to meaningless. Standard library functions such as strstr take a const pointer and return a non-const pointer derived from it, so compilers really can’t do anything other than treat it as documentation.
One thing that I think is interesting about plan 9 is that, in the later papers, they present a complicated song-and-dance of files you need to read from and write to in order to perform an operation–and then say that you probably don’t want to do that since it’s too complicated, and then give you a library to link to, so you can just call C functions instead of manipulating files.
Another limitation of the “everything is a file approach” is that it’s not necessarily evident how to use a subsystem. While it’s true that any program can just open
/my-subsystem/control, and start writing commands into it–the fact that/my-subsystem/controllooks like a file isn’t so useful, if you don’t know what commands to write into it!UNIX and Plan 9 have drawn favorable comparisons to Smalltalk. In many ways, Plan 9 feels like UNIX, but even more Smalltalk-y. The main difference is that the core operation in Smalltalk is sending a message, and messages can have customized names and meanings, which powers tools like the Smalltalk browser, which let you explore what messages an object will respond to. Plan 9 only exposes pre-defined messages like “open file,” “write to file”, etc.
One really cool thing that Plan 9 does, which I think is useful for both programming systems and operating systems, is that namespacing removed “ambient capability.” There’s no global “connect to the network” thing, for example. If you can connect to the network, it’s because the parent of your process has given you a file(ish), and you connect to the network by writing to that. Importantly, you have no clue if this is the “actual” network, or some sort of firewall that your parent process has set up. This is a powerful idea that underlies both Smalltalk and Plan 9–what matters is the messages that you send and the protocols that they respond to.
One modern-day system that’s working on this is Fuchsia. It has Plan 9-like namespacing, but a more Smalltalk-like messaging system. For example, here’s what the messages look like for a DNS lookup in Fuchsia. Your process can do a DNS lookup, if it’s given something that conforms to the
Lookupprotocol, but that could be the OS’s root DNS resolver, or something else.I’m not sure what you’re referring to – perhaps the internals of
dial(), but the benefits of a uniform file-based interface are there regardless of how it’s used. The uniformity around both system-visible operations and name resolution is the point.The key point is that it’s trivial to write interfaces, wrappers, and interposition layers that are completely agnostic to the RPCs being made. The network stack (
/net) can be exported and imported with completely generic tools. The mouse, keyboard, and display too – this is used for remote logins, giving a VNC-like experience from a 50 line shell script. The script itself doesn’t know anything about the device’s meta-protocol.You don’t even need to know what interfaces exist in order to transparently and fully proxy, inspect, or re-route them.
The Factotum paper recommends the C interface:
The Plan9 networking paper (so, yeah,
dial):The window system also suggests using a library instead of interacting directly with the files.
I agree that it’s easy to write servers which only proxy messages in the Plan9 system. I don’t know how that works in Fuchsia, but that’s really easy in Smalltalk. You can make an object which responds to all messages! You can then use whatever logic you want to ferry that message across the network, and then send that message to another object on another computer. I think the proxy server would actually look quite similar to the Plan9 version.
The Plan9 approach is much harder IMHO, if you want you actually manipulate the command stream. Consider for example, the
/net/dnsprotocoll:Let’s say I want all domains that exist to resolve to localhost, for some reason. In order to do this, the first thing I need to do is write a parser for this protocol. Not the hardest thing to do, but it still feels silly that this is a requirement.
The bigger issue is that this protocol is flawed.
TXTDNS records (at least) can contain newlines. My simple parser will need to be modified to handle this edge case (as well as, presumably, every other DNS server, judging from the description of this DNS protocol).Rather than pushing the parsing onto the user, leading to issues like this, in systems like Erlang, Smalltalk, and Fuchsia, the fact that messages are structured means, I think, that they’re much easier to work hit the ground running with.
The protocol is free form in theory, but it’s pretty much standardized in practice, at least for textual messages. The parsing is handled by the
tokenize()library call in userspace, or by theparsecmdcall in the kernel. The quoting is handled by the ‘%q’ format specifier. This handles edge cases pretty well.In practice, I often need less code and effort to deal with this than to write code against most other RPC libraries. It just isn’t a pain point.
A similar thing happens in Linux (raw system interfaces are too complicated to use, or are undocumented, so you use a C library interface instead that provides a higher level abstraction). A mild case of this is the new
io_uringsystem call, which is cumbersome to use, and periodically extended, so the author of the system call also maintains a C libraryliburing. A more extreme case is the raw system interface to the GPU, which AFAIK is undocumented, specific to a particular GPU vendor, and unstable (subject to change). Your only real choice is to use a library interface like Vulkan or OpenGL.Does Fuchsia also work this way? The DNS lookup interface you linked to is provided by a library called “fuchsia.net”. How much work is that library doing in userspace, and what does the raw system call interface look like?
Fuchsia is designed around message passing and capabilities. The actual system calls are basically just there to let processes communicate with each other. The DNS lookup interface that I linked to is the lowest-level DNS interface. Below that, it’s implemented by talking to the kernel to set up the communication between the client process and the DNS server process (which implements the DNS lookup protocol).
This
and this
are actually two sides of the same coin.
The simple, file-like namespaces let API developers fully separate privilege control.
The E programming language is an example of a system which gives users complete privilege control, but without 9P’s hard-coded requirement that the only allowable verbs are file system operations. I think whether you have an extensible set of verbs is a distinct issue from whether your system has ambient capabilities.
This is a general theme I’ve noticed with Linux: All the stuff I want is basically there, it’s just way more complicated and annoying than I’d like it to be.
Another example: OpenBSD gets praised a lot (rightfully imo) for its “pledge” system. Linux has something to do the same job, seccomp-bpf, but it’s so hard to actually set up that its usage is mostly limited to a very small set of core services.
There is most of a decent capability-based OS buried somewhere in Linux, you can send around file descriptors for almost anything (processes, sockets, memory, even namespaces themselves!) and you have a lot of tools to eliminate ambient authority (which is of course still there by default).
edit: and of course Linux also has a subsystem literally named “capabilities” that has nothing to do with any of this and is just generally awful.
pledge for linux: https://justine.lol/pledge/
It’s close, but it doesn’t quite do what pledge does. For example, you can’t pledge the shell to avoid doing any network I/O, while allowing programs spawned by the shell to still work.
It feels right that permissions obey a hierarchy, but it turns out to be very useful to break that hierarchy. It’d be even better to do it deeper than just a parent/child relationship, but that gets really complicated, so that one layer is a reasonable tradeoff in practice.
What’s the point of not letting the shell directly access the internet, while still letting it spawn other processes that can access the internet? Or am I misunderstanding?
Imagine that you’ve managed to find some bug in your shell; you want to pivot it to a complete expoit, but you don’t have any syscalls that you can use. If you’re careful about locking things down, you may have unveiled only a few binaries in the path, so you can’t even run arbitrary programs. Now, not only do you have to find an exploit in the shell, you have to pivot it to execute a program that will do what you want, and maybe garble its arguments in a way that it will give you more access.
In a privsep daemon, you may also have a spawner process that primarily has the job of creating processes that do the work; that process needs very little permissions to work, but the workers may need to do more.
It’s not a panacea, but it makes your attacker’s life a lot harder, and makes it possible to ratchet down permissions far more tightly.
To be fair, Linux Anything is Poor Man‘s Plan 9 Anything. Just recently I saw Longhorn, which is basically 9P2000 for people with weird beards.
What you mean by Longhorn? When I google “Longhorn”, I see info on Windows Longhorn (codename for Vista)
-> https://longhorn.io
It’s worth noting that Unix is fundamentally incompatible with free use of per-process namespaces; you can’t safely have setuid in an environment where the user can arbitrarily change the filesystem environment that a setuid program is running in. Plan 9 deals with this problem by doing away with setuid, an option that’s not available to Linux.
Examining Plan 9 namespaces offers valuable insights into simplification in system design.
Please, stop praising Plan 9. Plan 9 is too simple. Plan 9’s authors created very elegant and simple set of abstractions, but when these abstractions prevent them from solving some problem, they simply refuse to solve that problem!!!! For example, Plan 9 doesn’t have proper “mv” between different directories (I wrote on this previously: https://lists.sr.ht/~sircmpwn/public-inbox/%3CCAPnZJGBejeA+9APimVY6+HH%3DfWZJfW%3DQXJaOEK-JhhESRP+cSg%40mail.gmail.com%3E ). As well as I understand, proper “mv” is incompatible with “elegant” Plan 9’s abstractions, so the authors simply decided to throw it away (??????).
Also, Plan 9 is written in special dialect of C, which is incompatible with proper C. Why the authors created special language? If they really decided to create new language, why they made it mostly similar to C? Why not to create some really innovative language, such as Rust? So, instead of choosing between “normal compatible C” and “real innovation”, they chose worse sides of both. Look at any of Plan 9 sources, for example: http://git.9front.org/plan9front/plan9front/30c5296f32b87d83529d772732726891e1261c9c/sys/src/cmd/mv.c/f.html . What you see is absolutely obsolete programming practices. Raw C string manipulation. Raw pointers. And all this in userspace. I understand why C is used in kernels. But user space?
Plan 9’s abstractions make writing perfomant software very difficult (if possible), as noted here: https://yotam.net/posts/linux-namespaces-are-a-poor-mans-plan9-namespaces/#fn:3 in footnote 3.
Plan 9 is based on writing text commands in special text files. This is absolutely untyped protocol. It ignores all innovation based on type systems.
Compare this with true revolutionary Fuchsia OS. Fuchsia authors care about performance. “Simple and slow” is not their method. (Very small) kernel is written in C (the authors had reasons to do so), but most of userspace is written in Rust. Communication is based on typed Fuchsia IPC protocol
Because when they initially made C, they realized they got things wrong, and changed them.
And yet, it tends to perform better on many basic system tasks. Obviously not everything – Linux has had a lot of optimization work put into it, and having a monolithic kernel and VDSOs instead of userspace file servers for some services cases gives a pretty big advantage to Linux – but Plan 9 keeps up surprisingly well;
Small examples: pipes are faster on 9front than on Linux, as is process spawning or task switching. Measure it yourself and see – in practice, performance tends to be quite acceptable. The last time I checked, on comparable hardware (recent Ryzen), execing a binary was about 10x faster on Plan 9 than on Unix.
It’s not because of any real effort in this space, Simplifying things tends to lead to less work being done, which leads to reasonable performance.
Uhm, you know that Plan 9 is like 30 years older than Fuchsia?
Yes. I that is why I wonder why people still praise obsolete OS Plan 9
Try it and find out :). I wondered why people still praised it, myself, until I gave it a go.
FQA 4 - 9front Installation Guide
It runs well directly on certain hardware (esp. ThinkPads) but also in QEMU on almost all modern OSs.
Having used Plan 9 recently, one of its most notable characteristics is its blistering performance - it’s far more responsive than any mainstream desktop or mobile OS I’ve used in recent years, even when running on ancient hardware or in emulation.
I will agree that it makes writing efficient software difficult or impossible. But, the level of simplicity means that it’s still orders of magnitude more responsive than any Web or Electron app, and usually faster than a native app in a modern OS.
The first couple editions of Plan 9 were written in a new language called Alef: https://en.wikipedia.org/wiki/Alef_(programming_language)
Basically, programming language tech in the early/mid 90’s wasn’t up to the task of making something Better Enough To Be Worth It, at least as far as the Plan 9 team knew.
Hmmm. I’m not sure that’s true; I think that the real issue was that the Plan 9 team fundamentally disagreed with the philosophy of the things that were in my opinion Better Enough To Be Worth It - Lisp and Smalltalk.
They didn’t see those languages, and their approach to systems, as even being Better, let alone Worth It.
This may also have been related (though I don’t know which way around the relationship points) to Bell Labs’ plan to target embedded development.
Addition: Plan 9’s C dialect ignores “const” specifier!!!!! This means that the authors managed to make the language, which is even more unsafe than C!!!! (How this was possible? :) ). They throw away C’s const-correctness…
Works for me:
This is warning, not an error.
(But yes, okay, it seems I have old information. I get this info from original Thompson’s description of that C dialect. It seems the authors added this warning since that time.)
That’s probably the right call anyway,
constin C is next to meaningless. Standard library functions such as strstr take a const pointer and return a non-const pointer derived from it, so compilers really can’t do anything other than treat it as documentation.