For an in-memory data system, you need to be able to:
a) change the memory layout - choosing array of struct / struct of arrays topologies, and which fields are peeled off to cold structs
b) evolve the schema - add new fields, without necessarily having to change every bit of code
Being able to change all this means we can lay out the data based on the current performance characteristics of the code.
This is all a flexible storage layer, akin to index definition.
The layer on top, querying, and mutation - are separate.
The question is - how close can you bring an in-memory DB to raw structs/arrays.
“I have generally found that most computer science was done by 1970, and if you think you’ve come up with a new clever idea, probably check the 60s again.”
This quote goes hard. It’s also been my impression every time I’ve read computer science stuff from that time.
And I think a similar term gained popularity, maybe “Data oriented programming”?
Similar but different.
The article here is good, but there was more to Mike Acton’s viewpoint than just keeping similar data nearby in memory. He got into columnar, for example, which evolved into all the “struct of arrays vs array of structs” conversations.
So the article is interesting and on point imo but doesn’t go far enough. If you recognize your app needs to sling data before the CPU in the most efficient way possible, you’d do more to your memory layout than just be more relational with it.
The key is flexibility.
For an in-memory data system, you need to be able to: a) change the memory layout - choosing array of struct / struct of arrays topologies, and which fields are peeled off to cold structs b) evolve the schema - add new fields, without necessarily having to change every bit of code
Being able to change all this means we can lay out the data based on the current performance characteristics of the code.
This is all a flexible storage layer, akin to index definition.
The layer on top, querying, and mutation - are separate.
The question is - how close can you bring an in-memory DB to raw structs/arrays.
“I have generally found that most computer science was done by 1970, and if you think you’ve come up with a new clever idea, probably check the 60s again.”
This quote goes hard. It’s also been my impression every time I’ve read computer science stuff from that time.
Fabian’s DOD book is one of my favorite introductions to the relational model, funnily.
So DoD is a term I associate with Mike Acton and this talk: https://www.youtube.com/watch?v=rX0ItVEVjHc
And I think a similar term gained popularity, maybe “Data oriented programming”?
Similar but different.
The article here is good, but there was more to Mike Acton’s viewpoint than just keeping similar data nearby in memory. He got into columnar, for example, which evolved into all the “struct of arrays vs array of structs” conversations.
So the article is interesting and on point imo but doesn’t go far enough. If you recognize your app needs to sling data before the CPU in the most efficient way possible, you’d do more to your memory layout than just be more relational with it.