I also agree CRDTs are the future, but not for any reason as specific as the ones in the article. Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively. And certainly to build reliable systems. The abstraction of a single, global, logical truth is so nice and tidy and appealing, but it becomes so leaky that I think all successful systems for distributed state will abandon it beyond a certain scale.
Additional thoughts on large-scale distributed state and CRDTs, in the context of my current work at Fastly — an InfoQ interview, and a state at the edge blog post.
(I posted this same comment earlier in the HN discussion of the article)
CRDTs seem very promising, but at least when it comes to text editing, CRDTs still have a long way to go. The most exciting work in this area is being done by Ink&Switch [0]. They have a number of interesting real-world app prototypes based on CRDTs.
An interesting case where CRDTs failed is Xi-editor, where they tried to use CRDTs as the basis for a plugin system [1,2].
One of the biggest problems with CRDTs is the overhead needed to keep track of the full document history. The automerge [3] project has been working on efficient compression of CRDTs for JSON datatypes.
The idea of monotonic updates is really appealing at first, but I was disappointed when I realized there’s no good solution to handle deletions. Tombstones, to me, seem like kind of a hack, albeit a necessary one. Practically, CRDTs aren’t the silver bullet they might seem like at first.
Another lesson learned is that when ten people are editing the same paragraph, there’s not really a right answer. I think the key to implementing CRDTs is doing it at the correct level of granularity.
ProseMirror intentionally chose NOT to use CRDTs [4].
That ProseMirror post seems to be about operational transforms, not CRDTs. The argument is pretty much the same though: it’s easier to collaborate via a central server.
Great piece! I really appreciate the deep look at an area of computer science that I don’t know much about. I also really liked how the author motivates it by tying it to political and philosophical concerns.
Anybody who says , “I was wrong” gets a small automatic mental notch of trust and interest from me…. Conversely anybody who claims they are never wrong, gets a large mental red flag of distrust and disinterest from me.
I agree, which is why I think making “I was wrong” a part of the title is psychological manipulation.. The first question should then be “who are you?” why should your word hold weight, a lot of the time the “I was wrong” is just a status play by a clever meta-gamer who is still wrong in his (presumably) second attempt.
This time however the person has a nice followup to the“who are you” and I believe he’s right that lattice based datatypes are the (democratic) future.
Meta: this is a wonderful comment to remind all of us that 1) the internet is full of people playing games and meta-games, and 2) it doesn’t matter whether you’re doing that or not, some people will assume you are.
When I first got online, I’d make all sorts of jokes: self-deprecating, random puns, and so forth. I knew that I used humor as a way to lighten the mood. It took me some time to realize that 99.9999% of the world didn’t know me and assumed my jokes were manipulative (They were, of course, just not in the way they took them).
Recently, some friends on FB asked me to do a synopsis of what I’ve learned as a layman studying CV-19. Studying it was a hobby, so I did the best I could, prefacing my essay with “I am just some random guy online. I know nothing. Please do not take this as any sort of professional advice” I felt that was the most honest I could be. When I posed it, people came by and publicly wondered why I was so humble and claiming to know nothing. It was obviously a manipulative play on my part!
If you take this shit too seriously it becomes a world of mirrors. I’ve learned to give it a good faith effort, say “I don’t know”, “I was wrong”, or “I’m just some random internet clown” and move on. This commenter was happy with the followup to the essay. That established the bona fides from them. But in general there’s no universal answer to this conundrum. No matter what you create, you can be assured that some folks will view you in a negative light. It’s not you or them, it’s just a result of having so many hundreds of millions of people online, all with different motives.
I mean, on many websites and communities, definitely. But, for me and myself, I try to engage with people on the internet sincerely, unless that make it actively hostile to do so.
I generally think that this site, for example, hasn’t suffered from too much irony poison, but that could just be my impression.
I have some friends that have just given up on serious conversation at all. They feel the well is so poisoned, the net has become a game of everybody speaking with insincerity.
That’s a sad way to be. I’m glad there are only a few folks I know like this. I can understand why they’ve given up, but it still looks to me like it’s worth giving honesty your best shot.
I tried to post a response but it didn’t show up? My phone crashed at the same time so maybe that’s the cause..
Roughly two things to say:
Re poison; I think identity is the solution, but deciding how we structure identity on the internet is the political problem of this century so lets leave that aside for now.
But I mostly just want to clarify my stance, “I was wrong” in a headline, implies “and that is out of the ordinary” — so it’s an appeal to authority. Now I expect to see that claim backed up immediately or I don’t trust this person to be a careful communicator.
In this case that was exactly what happened and so the headline had the intended effect. Nevertheless I think any status play should be carefully evaluated.. or you’ll be influenced by all sorts of “experts” while you wade through a sea of platitudes. Mine included.
On a somewhat unrelated note: from my perspective blogging is impossible. Since I can’t know who the reader is I cannot back up my statements — people playing different language games with similar assumptions (but directed at various targets) means choosing sides based on how you argue your point (which any one of the groups would perhaps accept if you argue in their language).
The author can bypass alot of that by establishing himself as an authority on the subject and staying high level. A great strategy, therefore very popular, therefore people get very jaded by common patterns being misused…
On a somewhat unrelated note: from my perspective blogging is impossible
I’ve been on internet forums ever since there was an internet.
You’re on to something there. As the crowd size grew, I found myself either commenting/blogging longer and longer, trying to catch and prevent any misunderstanding … or just giving up and not commenting. So many discussions are now impossible to have.
Blogging is a good example of a medium where people can drop in, grab a sentence or two, and then misconstrue it to mean/imply anything they want. If you try to clarify? They’ll just say you ninja-edited your blog. Text really sucks for difficult conversations, in many ways.
This is a tough and non-trivial problem. I’m trying to switch to video. I suck at video, but it’s much more difficult to take it out of context with low effort and at least the folks watching can get a higher bandwidth by watching body language, inflection, etc.
I find the most disturbing content I consume to be people who claim to be experts, mean well, are trying to help folks, and give out terrible advice. There’s nothing to do but walk on and ignore it, but whenever I see it I feel as if a great harm is being done. It’s the kind of thing that face-to-face conversations were made for, imo.
We are still only just beginning to understand their implications, lattices will be key in distributed computing all the way from cpu cores coordinating in a single “computer” to global scale networks converging political organizations.
We are still only just beginning to understand their implications, lattices will be key in distributed computing all the way from cpu cores coordinating in a single “computer” to global scale networks converging political organizations.
Could you please pick a concrete example instead of a sentence that does not have any concrete facts or statements.
Well the concrete facts are the definitions of ‘lattice’ and ‘convergence’.
A lattice is the ‘loosest’ (meaning requiring fewest assumptions) hierarchical structure that has guaranteed convergence (there is some nuance but that is the subject of lattice theory).
The problems I mentioned were a tounge in cheek reference to ‘global state’.
No need to be mean just because you don’t understand what I was saying.
In 1948, the transistor had been invented, but it was not yet the present. CRDTs are in their infancy now just like the lump of germanium at Bell Labs back then.
I think the mathematical foundation underpinning CRDTs are very old. CRDTs already used in places where these have legitimate use cases. For me this is more of teenage level usage at the very least.
Nobody’s “right” about the future until it isn’t the future anymore.
I mean, I get it, it’s an idiom for talking about what’s the better way. The unspoken and unquestioned assumption is that the future is inevitably going to be better than the present. That’s a nice faith to have. It’s harder to maintain, as a naïve faith, when you start studying the past. Perhaps that helps explain why so few of us do.
Good article. However, I keep seeing articles about CRDTs for live collaborative document editing, and I’m not sure this is such an important application. It’s clearly a very hard, very interesting technical problem, but is it that significant in real usage? I’ve only rarely edited a shared doc at the same time as someone else. I use Google Docs, but nearly always I’m writing on my own, or spectating someone else’s changes, or commenting in the margin, none of which require this type of CRDT.
The word “demo-ware” comes to mind. Much like Wave itself, live shared editing has a high gee-whiz factor that (i think) outweighs its actual usefulness.
I’m much more interested in systems like Scuttlebutt and Dat that allow people to publish and update content they own. Collaboration can come about through commenting and annotation.
I also agree CRDTs are the future, but not for any reason as specific as the ones in the article. Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively. And certainly to build reliable systems. The abstraction of a single, global, logical truth is so nice and tidy and appealing, but it becomes so leaky that I think all successful systems for distributed state will abandon it beyond a certain scale.
Additional thoughts on large-scale distributed state and CRDTs, in the context of my current work at Fastly — an InfoQ interview, and a state at the edge blog post.
(I posted this same comment earlier in the HN discussion of the article)
CRDTs seem very promising, but at least when it comes to text editing, CRDTs still have a long way to go. The most exciting work in this area is being done by Ink&Switch [0]. They have a number of interesting real-world app prototypes based on CRDTs.
An interesting case where CRDTs failed is Xi-editor, where they tried to use CRDTs as the basis for a plugin system [1,2].
One of the biggest problems with CRDTs is the overhead needed to keep track of the full document history. The automerge [3] project has been working on efficient compression of CRDTs for JSON datatypes.
The idea of monotonic updates is really appealing at first, but I was disappointed when I realized there’s no good solution to handle deletions. Tombstones, to me, seem like kind of a hack, albeit a necessary one. Practically, CRDTs aren’t the silver bullet they might seem like at first.
Another lesson learned is that when ten people are editing the same paragraph, there’s not really a right answer. I think the key to implementing CRDTs is doing it at the correct level of granularity.
ProseMirror intentionally chose NOT to use CRDTs [4].
Some more good references are [5,6,7]
[0] https://inkandswitch.com/
[1] https://github.com/xi-editor/xi-editor/issues/1187#issuecomm…
[2] https://news.ycombinator.com/item?id=19886883
[3] https://github.com/automerge and https://github.com/automerge/pushpin
[4] https://marijnhaverbeke.nl/blog/collaborative-editing.html
[5] Kleppmann 2020, “CRDTs: The Hard Parts” https://www.youtube.com/watch?v=x7drE24geUw with HN Discussion: https://news.ycombinator.com/item?id=23802208
[6] Kleppmann 2019, “Interleaving Anomalies in Text Editors” https://martin.kleppmann.com/papers/interleaving-papoc19.pdf
[7] https://abishov.com/xi-editor/docs/crdt-details.html
Maybe, but it’s a hack that’s needed for deletion in a lot of hash table algorithms too.
You can do markdown links here.
That ProseMirror post seems to be about operational transforms, not CRDTs. The argument is pretty much the same though: it’s easier to collaborate via a central server.
Oops, I posted the wrong link. See this one about CodeMirror: https://marijnhaverbeke.nl/blog/collaborative-editing-cm.html
Great piece! I really appreciate the deep look at an area of computer science that I don’t know much about. I also really liked how the author motivates it by tying it to political and philosophical concerns.
Anybody who says , “I was wrong” gets a small automatic mental notch of trust and interest from me…. Conversely anybody who claims they are never wrong, gets a large mental red flag of distrust and disinterest from me.
I agree, which is why I think making “I was wrong” a part of the title is psychological manipulation.. The first question should then be “who are you?” why should your word hold weight, a lot of the time the “I was wrong” is just a status play by a clever meta-gamer who is still wrong in his (presumably) second attempt.
This time however the person has a nice followup to the“who are you” and I believe he’s right that lattice based datatypes are the (democratic) future.
Meta: this is a wonderful comment to remind all of us that 1) the internet is full of people playing games and meta-games, and 2) it doesn’t matter whether you’re doing that or not, some people will assume you are.
When I first got online, I’d make all sorts of jokes: self-deprecating, random puns, and so forth. I knew that I used humor as a way to lighten the mood. It took me some time to realize that 99.9999% of the world didn’t know me and assumed my jokes were manipulative (They were, of course, just not in the way they took them).
Recently, some friends on FB asked me to do a synopsis of what I’ve learned as a layman studying CV-19. Studying it was a hobby, so I did the best I could, prefacing my essay with “I am just some random guy online. I know nothing. Please do not take this as any sort of professional advice” I felt that was the most honest I could be. When I posed it, people came by and publicly wondered why I was so humble and claiming to know nothing. It was obviously a manipulative play on my part!
If you take this shit too seriously it becomes a world of mirrors. I’ve learned to give it a good faith effort, say “I don’t know”, “I was wrong”, or “I’m just some random internet clown” and move on. This commenter was happy with the followup to the essay. That established the bona fides from them. But in general there’s no universal answer to this conundrum. No matter what you create, you can be assured that some folks will view you in a negative light. It’s not you or them, it’s just a result of having so many hundreds of millions of people online, all with different motives.
The internet is so irony poisoned, I assume sincerity is irony and irony is sincerity. Makes it hard to assume good faith, even when we should.
I don’t have that overall impression myself.
I mean, on many websites and communities, definitely. But, for me and myself, I try to engage with people on the internet sincerely, unless that make it actively hostile to do so.
I generally think that this site, for example, hasn’t suffered from too much irony poison, but that could just be my impression.
I have some friends that have just given up on serious conversation at all. They feel the well is so poisoned, the net has become a game of everybody speaking with insincerity.
That’s a sad way to be. I’m glad there are only a few folks I know like this. I can understand why they’ve given up, but it still looks to me like it’s worth giving honesty your best shot.
There’s an algorithm for that, you know.
I tried to post a response but it didn’t show up? My phone crashed at the same time so maybe that’s the cause..
Roughly two things to say:
Re poison; I think identity is the solution, but deciding how we structure identity on the internet is the political problem of this century so lets leave that aside for now.
But I mostly just want to clarify my stance, “I was wrong” in a headline, implies “and that is out of the ordinary” — so it’s an appeal to authority. Now I expect to see that claim backed up immediately or I don’t trust this person to be a careful communicator.
In this case that was exactly what happened and so the headline had the intended effect. Nevertheless I think any status play should be carefully evaluated.. or you’ll be influenced by all sorts of “experts” while you wade through a sea of platitudes. Mine included.
On a somewhat unrelated note: from my perspective blogging is impossible. Since I can’t know who the reader is I cannot back up my statements — people playing different language games with similar assumptions (but directed at various targets) means choosing sides based on how you argue your point (which any one of the groups would perhaps accept if you argue in their language).
The author can bypass alot of that by establishing himself as an authority on the subject and staying high level. A great strategy, therefore very popular, therefore people get very jaded by common patterns being misused…
I’ve been on internet forums ever since there was an internet.
You’re on to something there. As the crowd size grew, I found myself either commenting/blogging longer and longer, trying to catch and prevent any misunderstanding … or just giving up and not commenting. So many discussions are now impossible to have.
Blogging is a good example of a medium where people can drop in, grab a sentence or two, and then misconstrue it to mean/imply anything they want. If you try to clarify? They’ll just say you ninja-edited your blog. Text really sucks for difficult conversations, in many ways.
This is a tough and non-trivial problem. I’m trying to switch to video. I suck at video, but it’s much more difficult to take it out of context with low effort and at least the folks watching can get a higher bandwidth by watching body language, inflection, etc.
I find the most disturbing content I consume to be people who claim to be experts, mean well, are trying to help folks, and give out terrible advice. There’s nothing to do but walk on and ignore it, but whenever I see it I feel as if a great harm is being done. It’s the kind of thing that face-to-face conversations were made for, imo.
I am not sure in what sense CRDTs the future. It appears to me that CRDTs are the present. This is a small piece of technology that is used in distributed systems. One example: https://riak.com/products/riak-kv/riak-distributed-data-types/index.html?p=10947.html
I am not sure why it is the future.
You may wish to read the actual article. The article is specifically talking about collaborative document editing.
Yet the title is about CRDTs.
We are still only just beginning to understand their implications, lattices will be key in distributed computing all the way from cpu cores coordinating in a single “computer” to global scale networks converging political organizations.
Could you please pick a concrete example instead of a sentence that does not have any concrete facts or statements.
Well the concrete facts are the definitions of ‘lattice’ and ‘convergence’.
A lattice is the ‘loosest’ (meaning requiring fewest assumptions) hierarchical structure that has guaranteed convergence (there is some nuance but that is the subject of lattice theory).
The problems I mentioned were a tounge in cheek reference to ‘global state’.
No need to be mean just because you don’t understand what I was saying.
In 1948, the transistor had been invented, but it was not yet the present. CRDTs are in their infancy now just like the lump of germanium at Bell Labs back then.
I think the mathematical foundation underpinning CRDTs are very old. CRDTs already used in places where these have legitimate use cases. For me this is more of teenage level usage at the very least.
Nobody’s “right” about the future until it isn’t the future anymore.
I mean, I get it, it’s an idiom for talking about what’s the better way. The unspoken and unquestioned assumption is that the future is inevitably going to be better than the present. That’s a nice faith to have. It’s harder to maintain, as a naïve faith, when you start studying the past. Perhaps that helps explain why so few of us do.
Good article. However, I keep seeing articles about CRDTs for live collaborative document editing, and I’m not sure this is such an important application. It’s clearly a very hard, very interesting technical problem, but is it that significant in real usage? I’ve only rarely edited a shared doc at the same time as someone else. I use Google Docs, but nearly always I’m writing on my own, or spectating someone else’s changes, or commenting in the margin, none of which require this type of CRDT.
The word “demo-ware” comes to mind. Much like Wave itself, live shared editing has a high gee-whiz factor that (i think) outweighs its actual usefulness.
I’m much more interested in systems like Scuttlebutt and Dat that allow people to publish and update content they own. Collaboration can come about through commenting and annotation.
Is Notion.so kind of the new Google Wave?