There was a lot of activity in the 1990s around rethinking the relationship between applications and storage. I still have a whole bookshelf of material about “persistent object systems”, a topic that used to be big enough to have its own conference but which has now completely disappeared from view.
Some of these ideas did ship commercially. There’s the obvious example of IBM OS/400 (quite a deep dive into the idea). Several filesystems appeared around this time with first-class attributes (BeOS, HFS, NTFS, etc.). None of them developed serious indexing or transaction features, though.
In 1995 I was working on the second release of Newton OS, which — unlike any other mass market OS I can think of — took this seriously enough that it had no filesystem, even internally, just a flat transactional blob store with an object serialization and indexing layer on top. Several years later I worked on WinFS, which attempted to merge SQL Server with NTFS (that did not go well). It turns out filesystems and databases look the same from a philosophical perspective, but actually have very different expectations from clients.
At this point everyone seems to have converged on a combination of (1) filesystems that don’t blow up with lots of little files, (2) a bunch of data in SQLite files, and (3) asynchronous indexing. You can find stuff a lot better, but applications are still as siloed as ever.
By bookshelf do you mean actual books? That sounds like an interesting collection for folk around here to reference. Any chance we can bother you to catalog the titles here or on librarything or something?
Thanks to Claude, that’s not as much trouble as it sounds! But good luck finding any of these now…
“Object-Oriented Concepts, Databases, and Applications” edited by Kim Lochovsky, published by ACM Press
“Persistent Object Systems, Newcastle, Australia 1989” edited by Rosenberg and Koch, Springer Workshops in Computing
“Advances in Object-Oriented Database Systems” edited by Dittrich (Springer Lecture Notes in Computer Science #334)
“Object-Oriented Database Programming” by Alagić, Springer Monographs (1989)
“Computer Systems with a Very Large Address Space and Garbage Collection” by Bishop (MIT/LCS/TR-178)
“Implementing Persistent Object Bases: Principles and Practice” edited by Dearle, Shaw, Zdonik, labeled as “The Fourth International Workshop on Persistent Object Systems”
“Database Programming Languages: Second International Workshop” by Richard Hull, Ron Morrison, David Stemple, published by Morgan Kaufmann (1989)
“Data Types and Persistence”, Atkinsin, Buneman, Morrison (Eds.), Springer Topics in Information Systems
“Readings in Object-Oriented Database Systems”, Zdonik, Maier (Eds.), Morgan Kaufmann
“Fifth international workshop on object orientation in operating systems”, Cabrera, Islam (Eds.), IEEE Computer Society, 1996
Probably the most inspirational things to me then were Peter Bishop’s doctoral thesis from 1977 (that MIT TR-178) and Eliot Moss’s work on persistent objects at U Mass (his papers are here).
I forgot two other important inspirations: Symbolics Statice (which I can’t even seem to find a description of) and the C++ product the same people made, ObjectStore.
which we may imagine results in less overall code, and more reuse.
Though the classic problem is that any bug where the application corrupts state, result in a bug in persisted state.
It breaks the “reboot computer” or “restart application” method of solving problems / working around bugs, which is honestly the most reliable and timeless one.
It also reminds me of the arguments around RPC – whether network protocols should be tightly coupled to the programming language, or more loosely coupled. Compared to what was proposed in the 90’s and such, it looks like looser coupling is what worked.
Most of these systems weren’t “passive” single-level stores like a huge virtual memory. There was some concept of committing or rolling back transactional changes to persistent objects, so the post-crash situation is basically the same as any database.
Schema evolution, on the other hand, was a big consideration, and I don’t know if anybody found a great solution to that. It’s the same problem we have today with document databases. But the obvious place to address it is the (de)serialization layer, and part of the point of persistent objects was that you didn’t have one.
While normally I would take this opportunity go remind people interested in database-oriented operating systems to check out IBM i (you should!), I’ve covered it a lot already. Instead, it’s worth noting the social context of this talk, since a lot of people here probably aren’t familiar with it. The classic Mac OS was starting to begin its decline, but it had a developer culture of its own. It was decidedly not a plain text oriented OS, and instead totally graphical, lacking the concept of a command line. There was a focus on direct manipulation of objects, making them user accessible, and as the paper mentions, the resource fork could provide structured data that was easily editable with tools.
I would also be interesting in reading more about this. The main IBM site is very enterprise-focused, so I’m having a hard time making sense of what it is, what it does, and how it works
Frank Soltis “Inside the AS/400” or “Fortress Rochester” (3 editions of essentially the same book under those 2 titles, any of them are fine since the used price has gone up) are worth getting for anyone really into Operating Systems. It goes very deep into the conceptual and how everything fits together but not as deep as a UNIX internals book which at the very least have public headers to show off data structures and interfaces.
There was a lot of activity in the 1990s around rethinking the relationship between applications and storage. I still have a whole bookshelf of material about “persistent object systems”, a topic that used to be big enough to have its own conference but which has now completely disappeared from view.
Some of these ideas did ship commercially. There’s the obvious example of IBM OS/400 (quite a deep dive into the idea). Several filesystems appeared around this time with first-class attributes (BeOS, HFS, NTFS, etc.). None of them developed serious indexing or transaction features, though.
In 1995 I was working on the second release of Newton OS, which — unlike any other mass market OS I can think of — took this seriously enough that it had no filesystem, even internally, just a flat transactional blob store with an object serialization and indexing layer on top. Several years later I worked on WinFS, which attempted to merge SQL Server with NTFS (that did not go well). It turns out filesystems and databases look the same from a philosophical perspective, but actually have very different expectations from clients.
At this point everyone seems to have converged on a combination of (1) filesystems that don’t blow up with lots of little files, (2) a bunch of data in SQLite files, and (3) asynchronous indexing. You can find stuff a lot better, but applications are still as siloed as ever.
By bookshelf do you mean actual books? That sounds like an interesting collection for folk around here to reference. Any chance we can bother you to catalog the titles here or on librarything or something?
Thanks to Claude, that’s not as much trouble as it sounds! But good luck finding any of these now…
Probably the most inspirational things to me then were Peter Bishop’s doctoral thesis from 1977 (that MIT TR-178) and Eliot Moss’s work on persistent objects at U Mass (his papers are here).
I forgot two other important inspirations: Symbolics Statice (which I can’t even seem to find a description of) and the C++ product the same people made, ObjectStore.
The persistent object systems sounds a bit like various ideas around “single level stores” … https://en.wikipedia.org/wiki/Single-level_store
which we may imagine results in less overall code, and more reuse.
Though the classic problem is that any bug where the application corrupts state, result in a bug in persisted state.
It breaks the “reboot computer” or “restart application” method of solving problems / working around bugs, which is honestly the most reliable and timeless one.
It also reminds me of the arguments around RPC – whether network protocols should be tightly coupled to the programming language, or more loosely coupled. Compared to what was proposed in the 90’s and such, it looks like looser coupling is what worked.
Most of these systems weren’t “passive” single-level stores like a huge virtual memory. There was some concept of committing or rolling back transactional changes to persistent objects, so the post-crash situation is basically the same as any database.
Schema evolution, on the other hand, was a big consideration, and I don’t know if anybody found a great solution to that. It’s the same problem we have today with document databases. But the obvious place to address it is the (de)serialization layer, and part of the point of persistent objects was that you didn’t have one.
While normally I would take this opportunity go remind people interested in database-oriented operating systems to check out IBM i (you should!), I’ve covered it a lot already. Instead, it’s worth noting the social context of this talk, since a lot of people here probably aren’t familiar with it. The classic Mac OS was starting to begin its decline, but it had a developer culture of its own. It was decidedly not a plain text oriented OS, and instead totally graphical, lacking the concept of a command line. There was a focus on direct manipulation of objects, making them user accessible, and as the paper mentions, the resource fork could provide structured data that was easily editable with tools.
I would also be interesting in reading more about this. The main IBM site is very enterprise-focused, so I’m having a hard time making sense of what it is, what it does, and how it works
Frank Soltis “Inside the AS/400” or “Fortress Rochester” (3 editions of essentially the same book under those 2 titles, any of them are fine since the used price has gone up) are worth getting for anyone really into Operating Systems. It goes very deep into the conceptual and how everything fits together but not as deep as a UNIX internals book which at the very least have public headers to show off data structures and interfaces.
Everything I’ve ever read about IBM i has been interesting! Do you have pointers to where you’ve covered it?