1. 62
  1.  

  2. 28

    A number of the “wishes” in here were already in the BeOS, and of course now live on in Haiku:

    Configuration files and their various formats exist because the filesystem can’t save a structure. If it could, you could just save a given object or dictionary, with all its keys and values, and you’d be able to use them directly next time you’d needed them. … When a program needs to send data to other program, it doesn’t send a serialized version, messages are not sent as stream of bytes, but as structured messages, natively supported by the OS kernel.

    This mostly lines up with the “Messages and Ports” concepts in BeOS and Haiku.

    Programs as collections of addressable code blocks in a database … Once you publish them, you stop having to communicate using the old stream-based methods (sockets, files, …) - it suffices you to just return structured data.

    Since on Haiku, every window is its own thread, applications communicate even with themselves using messages; and if they allow it, any application can send them messages, and even ask what messages they accept.

    A database instead of a filesystem … I’m talking here about a generic structured system of storing data on media that supports data types, atomicity, indexing, transactions, journals and storage of arbitrarily structured data, including large blocks of purely binary data.

    BeOS and Haiku make extensive use of (typed) extended attributes, which are then indexed and can be queried. Haiku’s Email client utilizes this by storing each e-mail as its own file, with attributes like MAIL:from, MAIL:subject, etc. There are also attributes like BE:caret_position, which specifies where the caret last was when you closed a (text) file, so any editor can restore your position, no matter which one you last opened a file in.

    1. 6

      Since on Haiku, every window is its own thread, applications communicate even with themselves using messages; and if they allow it, any application can send them messages, and even ask what messages they accept.

      This is also the case, I believe, in Plan 9

      1. 2

        I’ll have to look into Haiku some day, thanks for the pointers.

      2. 19

        Another OS that does some of these (I haven’t dug into this fully, just skimming) is IBM i:

        • Objects are stored with structure in memory. Because everything exists in a single address space, you just pass pointers to objects, which also double as capabilities (!) since a hardware accelerated trusted translator enforces pointer acquisition. Single level storage means all objects have fixed addresses (where memory pages correspond to disk pages) which persist on cold reboot (!).

        • Command-line parameters to programs are typed; you can receive integers, strings, and pointers (which obsolete most of the need for environment variables or stdio). These are abstracted by commands, which provide consistent scripts and wrappers for type-safety, validation and documentation (it integrates fully in the context-sensitive help system where you can press F1 with your cursor on anything) on top of just shunting things into argv.

        • SQL is built on basic system primitives and given a privileged position on the system technically (much of DB2 is in the kernel) and culturally (almost all of the users use it for RDBMS-using applications)

        1. 16

          The Newton OS was based on some similar thinking to this, though not the networking part, because nobody had a network back then!

          In Newton OS you store the language-level objects (which are key/value containers kind of like JavaScript objects) directly in the not-a-filesystem object store and retrieve them (by content, not by name). Those stored objects could be executable — a function is just a simple object with a few slots, one of which is a blob of bytecode. We had some further ambitions for when the hardware got bigger that we didn’t get to realize.

          If you google for “persistent object systems” you can still see a bunch of the papers we were reading back then. It was a hot topic for a while.

          To quote my ancient overview paper:

          The object store has many advantages over a more traditional filesystem. Applications do not need to implement their own methods of translating objects between runtime form and persistent form, so they are smaller and easier to write. It is very easy for Newton applications to share data, since they all get the same high-level interface to it; we encourage developers to share their persistent data structures to foster the creation of highly cooperative applications.

          1. 4

            Thanks for reminding us of the Newton (and for your creation, NewtonScript). I really like the name for the object store… ‘soups’. I had a message pad 1000 and it was a system way ahead of its time.

            1. 2

              I’ve read Object-Oriented Database Systems by Elisa Bertino & Lorenzo Martino (ISBN 978020162439), so I have some idea about object databases of the 90’s.

              In Newton OS you store the language-level objects (which are key/value containers kind of like JavaScript objects) directly in the not-a-filesystem object store and retrieve them (by content, not by name). Those stored objects could be executable — a function is just a simple object with a few slots, one of which is a blob of bytecode. We had some further ambitions for when the hardware got bigger that we didn’t get to realize.

              That sounds kinda similar to Self. I’ll have to look into that.

              The Newton OS was based on some similar thinking to this, though not the networking part, because nobody had a network back then!

              Didn’t Newton used Telescript for this? Edit: hah, it didn’t, sorry.

              1. 4

                The NewtonScript data model and language were certainly influenced by Self. The object store was probably most strongly influenced by Eliot Moss’s work at UMass (e.g., Working with Persistent Objects: To Swizzle or Not to Swizzle). There was also a lot of pondering of various single-address-space research, but we never got to the point of using it. Well, now that I think about it, we did use a variation of it (not as clever) for 3rd party application packages.

            2. 13

              I already wrote a critique of parts of this article, on hacker news. I’m not entirely sure it’s going to get much visibility there (Because nobody there seems particularly interested in it), so I’ll repost it here, because I’m interested in how people respond to it:

              It’s probably worth the owner checking out the Lisp Machine, Oberon, and Inferon (A descendent of Plan9 where they improved over the original concept). They say Plan9 is a step in the right direction, but rather than attacking a specific aspect of Plan9, they critique how the concepts were imported into Linux, which seems inherently silly to me.

              It’s like critiquing a master craftman’s tools by observing how a master craftman uses his tools, and then critiquing how the apprentice uses those tools. Sure, some of that criticism of the concepts is valid, but they picked the worst implementation of those concepts to critique.

              One of the things mentioned in the critique is how the filesystem in question communicates the changes back to the user, and how the user is expected to know what parts of the filesystem to alter. While in the examples given, it does seem relatively obscure, the decry the fact that they had to check the manual for the position to write to.

              However, both of those would still be the case for the system that they talk about. Let’s talk about object orientated systems, which already exist, in limited (compared to the operating system) forms. You still have to pick up a manual to figure out what keys and functions are relevant to you. In Pharo (which is more or less the system they’re after!), you can inspect objects that exist and the methods that exist for them – however – often, similarily-named functions do different things, which also requires reading a manual. In addition, using these tools as a first-time user, I was overwhelmed by the number of functions available, most of which I could only guess at what their purpose was. Pharo integrates the manual into the system, but the manual is still there!

              Standardized error codes are given in structures passed back (As in Erlang) or in Exceptions, but there are problems with those too. You still have to figure out what those exceptions or errors mean, and there isn’t necessarily a standardized format for those errors. As a developer I’ve been recently been working on writing a Python api. As someone who has little experience with the intimate details of the Python language, I do not always know of the exceptions that exist that I am able to throw, I do not always know of which exceptions would be most appropriate. The same problem exists (obscure error values), but in a different form. There are existing libraries that I rely on that implement more exceptions, sometimes these aren’t documented, and even with the venerable Requests library, I have still had to crack open the objects that exist to find easier methods of passing data that the Requests library has, and uses, but does not document for external use.

              Let’s look at the windows hive database. That’s a database of keys for values in the operating system. As a random developer, would you be able to open it and figure out what it does? I wouldn’t. As a windows power user I often relyied on articles from howtogeek without really fully understanding what the keys were doing (Although because of my experience an a systems developer, I could guess, but only after the fact). Again, the same problem (“I know what I want to do, but I do not know how to do it”) is exposed in a different form, and the methods and practices of the Microsoft organization make that difficult hard to reach. Yet again, the same problems are there, but in a different form.

              I do agree that the shell should handle program arguments, a program could expose keys and values and a shell could read that and tell the user about it. I would be interested to see the problems that the Oil shell encounter in their implementation of an object-orientated shell.

              1. 2

                Understanding the meaning of symbols and navigating large sets of symbols are certainly problematic, but it’s separate from the problems of unstructured data and the redundant transformations, which is what the article is mostly focused on, I think.

                I also think that the first two problems (which you are talking about) could be mitigated to a large degree. They are not fundamental. Similarly to the problem of unstructured data, they are caused in large part by an unnecessary separation between symbols and their descriptions (and the descriptions of their relationships). Even here, it would probably be helpful to have a universal store that could incorporate a variety of objects along with their descriptions! The other cause is that nobody seems to have a particular incentive to provide decent descriptions. It appears to be cheaper to get an army of programmers to internalise them, breaking systems apart into a multitude of layers to mitigate the programmers’ limited capacity for memorisation.

                1. 1

                  As a random developer, would you be able to open it and figure out what it does?

                  No, but as a Windows developer you would. There’s nothing mysterious about the registry. It’s just a tool like any other.

                  1. 4

                    I think I meant, given a subset of the registry, would you be able to figure out what it does? Regardless, your comment assumes that the person is trained in the thing that they’re using, which is an assumption that didn’t hold for the article in question. Someone familiar with the linux /sys layout,or with GPIO ports, would already be familiar with the filesystem layout. The question is how easily can a developer figure out a subset of the system that they are not intimately familiar with.

                2. 13

                  Great article. A bit naive, but all manifestos are.

                  A big part of the problem is that we’ve been unable to agree on what a higher-level data representation would be. It has to be rich enough to represent everything people want to store, fast enough, scalable enough, portable enough, etc. There are a lot of features that are must-haves for some use cases, unnecessary for others: random-access, ACID, mutable-in-place, human-readable, extensible, secure, …

                  There are generic structured formats for media data that have become widely used: IFF (used by AIFF), QuickTime, MPEG. But they’re not useful for domains that need different features. Similarly for JSON and XML in web APIs, or YAML and TOML for configuration files. You’d never use MPEG format for a config file, or YAML to encode a 2GB video.

                  There’s a fake RFC somewhere that specifies “IP-XML”, with a full schema for encoding IP packets in XML. It’s a joke, of course.

                  1. 2

                    My thoughts as well. It would be nice to create the abstractions on top of the FS (or somewhere on OS level) that would make the author’s job of managing many small files easier - but those abstractions would not work that well on my use case.

                    They did raise a good point though - there’s very little (apparent to me) research in this sphere. While we can’t create a representation that works for everybody, we could create a few of them that work great for somebody specific. Figuring out a few of the most relevant ones and pushing them would solve some of the authors’ problems.

                  2. 8

                    I wish the author would have given an example of what he means by “structured data”. Yes, I could send raw structures across the network and that’s fine, until I have a program running on x86 needing to talk to a program running on SPARC and now I have to take care of issues of endianess and alignment. And what if one of the fields is an actual string? Sending a pointer across the network won’t work. So even if I’m talking x86 to x86, I still might need some serialization. I also don’t think the author realizes what ZeroMQ really gives you—it’s just a fancy wrapper around the various BSD socket calls—the user is still responsible for marshaling the data.

                    1. 2

                      until I have a program running on x86 needing to talk to a program running on SPARC and now I have to take care of issues of endianess and alignment.

                      Surely as it’s sent over the network, it would be converted to the standard Network Byte Order, no? I don’t have too much knowledge about networks, so that’s me asking a question, not me posing a question to make a point :’)

                      I feel that maybe Pike’s critique of the byte order fallacy might apply here too?

                      1. 5

                        If it’s an x86 system to x86 system (or rather, any other little endian system without alignment restrictions) there’s no real need to convert everything to network byte order (which is big endian by the way). Pike’s criticism of byte order only relates to reading/writing bytes from another system, which requires the code to know what endian to expect.

                        But if you are sending between systems of differing endianness, then yes, you need to decide on an ordering.

                    2. 8

                      The reason for UNIX-like OS’s having low level abstractions like byte streams, filesystems of blocks/inodes/directories and hundreds of other like things, was to retain the efficiencies of the low level machine exported to time multiplexed processes on a single processor, where you could build upon them whatever complexity you’d like.

                      Many OS types I knew from the 69’s-70’s never even saw a LISP Machine (LISPM) OS, nor had much contact with such systems that never had any interest in retaining low level abstractions, because those users were more interested in high level abstractions to be fleshed out rapidly so they could be used on the machine, and then as they were used, they would be refined to make them efficient enough.

                      We’re in a very different universe of multicore, cluster, and cloud, where you can buy all the cycles you want in the world for the nanoseconds you might need them. Waiting to code/debug/process/parallelize high to low level to run on a low-level OS takes more time than the high level abstraction itself needs to function. So there is “anti-efficiency” many face.

                      Why is this happening? Well … standards. Like POSIX. Now that we’ve finally made them work, they can’t be replaced, too much depends on them. And, unlike on LISPM’s, you can’t just rebuild the APIs on the fly to suite your need, because several billion others need to work with them at the same time, and they want to build there own differently for both good and bad reasons.

                      There’s no escape through metastandards either, because that evasion will be countered in turn by efficiency arguments (which are mostly specious because people work off of fear and not measured, pragmatic study because that takes considerable talent/skill/work, which often isn’t valued as much as it should be).

                      We are at a crossroads where we need something to break the logjam w/o being sucked back into to old standards, but since no one trusts anyone to do it, it never goes anywhere. Such is what happens when the free OS’s sucked all the air out of OS research decades ago because it was no longer necessary, because the truth of a common low level OS was all we would ever need.

                      Well guess what, we have it now, and by golly its nowhere near what we need for the future.

                      1. 1

                        Well guess what, we have it now, and by golly its nowhere near what we need for the future.

                        Its also nowhere near to what we need for now. I literally created (and still create) same abstractions over and over again just to get away from this.

                      2. 6

                        I wrote about this last year, and I enthusiastically agree with many of the points you have raised, and the examples you have given.

                        I’d like to add another way to look at the issue: Unix-like systems fundamentally lack a layer of abstraction around programs, mainly because of the shell. It makes little sense if the only interface is the shell, but imagine if instead of just parsing argv, programs could assume some structured format. The critical advantage would be that now you could reuse a program within a graphical environment, as one did in a shell. It’s interesting to note that this would just be a change in convention, not in technology. I’m sure a lot more could be improved if we thought critically about our primitives, than most people do now.

                        Another example would be “structured”-user interfaces (TUI, GUI). It seems quite popular in some circles to use curses interfaces to do some things, where I don’t necessarily see the advantage of doing it in a terminal emulator (browsing files with ranger, listening to music with ncmpcpp, …). It usually seems like some kind of a self-imposed challenge… Here again, a lack of abstraction duplicates the computational effort, since on the one hand each program has to draw it’s own UI – even if the logic is usually abstracted into libraries – and on the other hand, terminal emulators have to decode it. The UNIX Haters Handbook mentioned that ITS had a different approach (p. 112):

                        At the MIT AI Laboratory, a different solution was developed. Instead of teaching each application program how to display information on the user’s screen, these algorithms were built into the ITS operating system itself. A special input/output subsystem within the Lab’s ITS kernel kept track of every character displayed on the user’s screen and automatically handled the differences between different terminals. Adding a new kind of terminal only required teaching ITS the terminal’s screen size, control characters, and operating characteristics, and suddenly every existing application would work on the new terminal without modification.

                        Of course this too was limited, but again: with the right abstractions the same interface could have allowed different modes of presenting user interfaces. But I guess that’s the problem – the right abstractions seem to only appear in a posteriori. The only way out of this issue, that I see is to build systems that can handle and profit from a critical retrospective review.

                        1. 6

                          It’s pretty amazing that despite everyone running terminal emulators since at least X11 became a thing, termcap still stick around, a sort of reverse appendix with a ton of useless outdated configuration around the small kernel everyone actually uses.

                          1. 6

                            It usually seems like some kind of a self-imposed challenge…

                            Most TUIs can be operated without needing to look at the screen, not only are they faster in response time, but button-function binding is much faster to access and use as compared to navigating using a mouse. There’s a very good thread by gravislizard about this: https://threadreaderapp.com/thread/927593460642615296.html

                            There’s more, but a tl;dr of the point:

                            When computers used interfaces like these they were lightning fast, universally. I’ve used a lot of them. The library computers, the homebrew POS at my first job, even the machines at Fry’s Electronics use an interface like this. If you ever go to Fry’s, watch the employees use this style of UI. Blinding speed. They ricochet around like pros, even the new people. And I have to say that it’s solely because the mouse is not being used. Mice are bad. Mice are absolutely terrible. The reason mice are terrible is a matter of basic facts about human brains, hands, eyes and muscles. Hell, I think Jef Raskin covered it. Keyboards present fewer possible discrete options. Mice present a continuum. One can be operated blind; the other requires feedback. You cannot use a mouse without using your eyes to confirm everything. At my first job I rang people up without looking at the screen. POS software is designed VERY pragmatically. Typically there is no concept of “input focus”; all text goes to one field. In this UI, most likely at any given moment keyboard entry will ONLY go to the “Stock Number” field. When you walk up to this computer, you can rest assured of the following:

                            a) your input will go there

                            b) if not, pressing escape will fix it

                            If you don’t know the stock number, press F1 to be transported AWAY FROM THIS INTERFACE to look it up. this is critical. […]. Anyway. Once you find the stock number you want, the price shows up. Hit F3 to edit it. * to change quantity. Enter to add to invoice. In practice this is incredibly fast. You can’t even see it happening, because there’s no reason to except to verify after the fact. The operator doesn’t have to stop for every action and figure out what they’re doing time and time again. It’s all muscle memory. The cashiers hands are just a blur on the keys. They’re looking at the products, not the screen. And they get it right. Part of the reason for this is that the entire keyboard gets used, including the function keys. INCREDIBLY powerful, those.

                            God. God! What a tragedy, that we left them in the past. What a heinous crime that we forgot their value. Look at that UI up there. Look how many functions you can access AT ANY TIME. You can RELY on them. They will ALWAYS be there. In these systems, the screen was always dominated by a specific form that you could identify from across the room. […]. And the F-keys were always bound to that form. This allowed your /human brain/, a flawed but powerful tool, to use the computer INTUITIVELY. As you approach e.g. the cash register, you take one look at the screen and instantly your synapses wire themselves up for the F-key layout. If the register is on the stock lookup screen, your hands fall on the keyboard and automatically do what’s needed. Escape-escape to get to the login screen, fingers on the numpad hammer in your employee ID, F2 to cancel retail customer lookup. And now you’re ready to start keying in items. No “getting your bearings.” You don’t even stop talking to the customer. 1/10th of a second. To be clear, the web as I envision it does not look like what we have. Not one bit. It’s completely different. And it’s worth noting that HYPERTEXT, specifically, is best with a mouse in a lot of cases. Wikipedia would suck on keyboard. This is a FANTASTIC example of a mouse-optimal document. Any keyboard approach would be mediocre at best.,

                            1. 2

                              The sad state of desktop software is another issue, I was thinking more of /r/unixp**n of screenshot threads on imageboard -kinds of people.

                            2. 2

                              Another example would be “structured”-user interfaces (TUI, GUI). It seems quite popular in some circles to use curses interfaces to do some things, where I don’t necessarily see the advantage of doing it in a terminal emulator (browsing files with ranger, listening to music with ncmpcpp, …). It usually seems like some kind of a self-imposed challenge…

                              As the general UI situation is horrible, and Delphi/Lazarus have gone out of style, the easiest way to write a keyboard-controlled program quickly is to have a terminal emulator UI. Anything else requires even more effort, and for a file list or a music player it is not clear that extra effort will ever pay off for the author; the main information to display is text that doesn’t benefit much from different sizes.

                              One could do some Web UI, but then there is the overhead of a browser (definitely larger than a terminal emulator); for an actual GUI application, there is too much development overhead.