I don’t really see what use-case Redis Cluster satisfies. It’s not consistent. It’s not available. It can lose writes. Who actually wants those properties?
Playing fast and loose with data is a pretty common theme when you can realize significant performance advantages. The other question is, what kinds of writes does it lose? Imagine we’re caching user profile data in redis, and so we write to redis and journal to a persistent store at the same time. Someone accidentally submits two profile changes simultaneously that are practically the same, and one of our writes gets lost. This is probably OK–it doesn’t happen very frequently, and the cost is not that big–UIs usually don’t allow changes that fast, and we are used to expecting that this kind of behavior can lead to several different results.
Especially as since redis is primarily a cache, and was not designed as a persistent store, this seems like the right tradeoff for redis. Harvest and Yield has some neat ideas on how to reason about consistency and availability without just trying to choose either all A or all C.
I would agree with you if Redis Cluster chose Availability and said “we’ll lose writes”. But it doesn’t. If you’re on the minority side you’re unavailable. So in that case I don’t see what it’s offering as a cache.
This is totally speculative, but my guess is that they chose to disallow progress on the minority to avoid split-brain behavior, which is difficult to reason about, and also makes merging inconsistent results much more difficult.
But it’s not a consistent system since it uses async replication. So one is paying the cost of being consistent without the benefits. I find that kind of odd.
Because resolving two splits when they both accept writes willy-nilly is an insanely hard problem?
Also because one side won’t have access to correct data. A big win for eventually consistent systems is they are usually consistent within seconds unless a particular key sees a high volume of writes. Sacrificing a little short term consistency doesn’t imply they should also give zero fucks about long term consistency and openly allow split brain behavior.
The semantics of Redis Cluster are not consistent. You can get a piece of data that will never make sense.
My point is not that these problems are easy, it is that these problems are hard, and Redis Cluster has chosen semantics that don’t solve the problems well.
You find it’s odd that Redis Cluster uses strategies that reduce the number of inconsistencies in the database because it isn’t 100% consistent at every moment? Mitigation of inconsistencies is super important for any eventually consistent database.
Of course the problems are hard, but I disagree that Redis Cluster has bad semantics. In my opinion mitigating inconsistencies is a feature. What semantics would you prefer?
Redis Cluster chooses consistency in favor of availability. If you are on the minority side of a partition, you cannot modify data.
Redis Cluster chooses latency in favor of consistency. Replication is asynchronous.
Ok, so something bad has happened in your cluster. You read a piece of data: How do you determine if it is a valid piece of data? Given the async replication, writes can be lost.
In Riak or Cassandra you either use a data structure that is distribution friendly. Or you make your data immutable.
Redis Cluster is explicitly said the former is not an option, they don’t want the overhead. The reason they don’t want the overhead is because of the latter reason: you use Redis for mutable data.
So Redis Cluster has chosen semantics that are neither available nor consistent. I can’t count on uptime and I can’t even trust the data I get back. I don’t know of many problems that fit this use-case.
Firstly, you failed to answer my question. What semantics would you prefer?
I don’t know of many problems that fit this use-case.
Then think harder. Redis Cluster is this way because it doesn’t change the behavior Redis has always had. And last I checked, Redis has a ton of users.
Redis Cluster chooses consistency in favor of availability. If you are on the minority side of a partition, you cannot modify data.
Much like previous versions of Redis, where you cannot modify data if the master goes down.
Redis Cluster chooses latency in favor of consistency. Replication is asynchronous.
Much like previous versions of Redis, where replication is asynchronous.
Given the async replication, writes can be lost.
Much like previous versions of Redis, where updates are written to disk asynchronously, making it possible to lose writes.
So Redis Cluster has chosen semantics that are neither available nor consistent.
Name one distributed database that is fully available, or fully consistent. Like literally every other distributed database, Redis Cluster is a fuzzy blend of the two.
The Redis Cluster strategy and semantics are consistent with the existing behavior of Redis. Big surprise there.
The post actually specifically addresses this comment.
Moreover, I believe Redis Cluster could benefit from a special execution mode specifically designed for caching, where nodes accept writes to hash slots they are not in charge for, in order to stay available in a minority partition.
Cassandra has all those properties as well, e.g. it resolves eventual consistency with a last write wins strategy which can clobber legitimate writes. You have to design your application to deal with that. Redis Cluster also mitigates write loss by making the minority side of a partition unavailable, as said in the post.
Any time you make a database distributed, you make BIG BIG trade-offs. Right now, the only ways we know of to avoid data loss all involve huge taxes like dramatic latencies and client-side conflict resolution. Obviously, that kind of latency can’t be introduced into Redis, and conflict resolution doesn’t really fit in either.
I don’t really see what use-case Redis Cluster satisfies. It’s not consistent. It’s not available. It can lose writes. Who actually wants those properties?
Playing fast and loose with data is a pretty common theme when you can realize significant performance advantages. The other question is, what kinds of writes does it lose? Imagine we’re caching user profile data in redis, and so we write to redis and journal to a persistent store at the same time. Someone accidentally submits two profile changes simultaneously that are practically the same, and one of our writes gets lost. This is probably OK–it doesn’t happen very frequently, and the cost is not that big–UIs usually don’t allow changes that fast, and we are used to expecting that this kind of behavior can lead to several different results.
Especially as since redis is primarily a cache, and was not designed as a persistent store, this seems like the right tradeoff for redis. Harvest and Yield has some neat ideas on how to reason about consistency and availability without just trying to choose either all A or all C.
Not being designed as a persistent store doesn’t stop people from using it that way, though.
I would agree with you if Redis Cluster chose Availability and said “we’ll lose writes”. But it doesn’t. If you’re on the minority side you’re unavailable. So in that case I don’t see what it’s offering as a cache.
This is totally speculative, but my guess is that they chose to disallow progress on the minority to avoid split-brain behavior, which is difficult to reason about, and also makes merging inconsistent results much more difficult.
But it’s not a consistent system since it uses async replication. So one is paying the cost of being consistent without the benefits. I find that kind of odd.
Because resolving two splits when they both accept writes willy-nilly is an insanely hard problem?
Also because one side won’t have access to correct data. A big win for eventually consistent systems is they are usually consistent within seconds unless a particular key sees a high volume of writes. Sacrificing a little short term consistency doesn’t imply they should also give zero fucks about long term consistency and openly allow split brain behavior.
The semantics of Redis Cluster are not consistent. You can get a piece of data that will never make sense.
My point is not that these problems are easy, it is that these problems are hard, and Redis Cluster has chosen semantics that don’t solve the problems well.
You find it’s odd that Redis Cluster uses strategies that reduce the number of inconsistencies in the database because it isn’t 100% consistent at every moment? Mitigation of inconsistencies is super important for any eventually consistent database.
Of course the problems are hard, but I disagree that Redis Cluster has bad semantics. In my opinion mitigating inconsistencies is a feature. What semantics would you prefer?
Redis Cluster chooses consistency in favor of availability. If you are on the minority side of a partition, you cannot modify data.
Redis Cluster chooses latency in favor of consistency. Replication is asynchronous.
Ok, so something bad has happened in your cluster. You read a piece of data: How do you determine if it is a valid piece of data? Given the async replication, writes can be lost.
In Riak or Cassandra you either use a data structure that is distribution friendly. Or you make your data immutable.
Redis Cluster is explicitly said the former is not an option, they don’t want the overhead. The reason they don’t want the overhead is because of the latter reason: you use Redis for mutable data.
So Redis Cluster has chosen semantics that are neither available nor consistent. I can’t count on uptime and I can’t even trust the data I get back. I don’t know of many problems that fit this use-case.
Firstly, you failed to answer my question. What semantics would you prefer?
Then think harder. Redis Cluster is this way because it doesn’t change the behavior Redis has always had. And last I checked, Redis has a ton of users.
Much like previous versions of Redis, where you cannot modify data if the master goes down.
Much like previous versions of Redis, where replication is asynchronous.
Much like previous versions of Redis, where updates are written to disk asynchronously, making it possible to lose writes.
Name one distributed database that is fully available, or fully consistent. Like literally every other distributed database, Redis Cluster is a fuzzy blend of the two.
The Redis Cluster strategy and semantics are consistent with the existing behavior of Redis. Big surprise there.
The post actually specifically addresses this comment.
Cassandra has all those properties as well, e.g. it resolves eventual consistency with a last write wins strategy which can clobber legitimate writes. You have to design your application to deal with that. Redis Cluster also mitigates write loss by making the minority side of a partition unavailable, as said in the post.
Any time you make a database distributed, you make BIG BIG trade-offs. Right now, the only ways we know of to avoid data loss all involve huge taxes like dramatic latencies and client-side conflict resolution. Obviously, that kind of latency can’t be introduced into Redis, and conflict resolution doesn’t really fit in either.
I’ll be curious to see its behavior under Jepsen (or like) testing.