On a table you would find Jepsen, essentially the equivalent of Yelp, ideally meant to keep restaurants honest, generally correct, but ultimately not that different from other tech companies,
This is a really strange take on the role that Jepsen plays in the ecosystem. Kyle isn’t motivated by any of the factors that make Yelp so pathological; if we’re going to place Jepsen into the restaurant metaphor, I’d say they’re the health inspector. Can you say more?
Hi Peter, it’s part of the truly ranty part of the post (and explicitly marked as such), so I wouldn’t take it too seriously, but in this one blog post dedicated to describing my experience with Redis, I wanted to call out all the organizations that in one way or another I feel have not done justice to Redis, even if not directly and without monetary reasons to do so.
The history between Redis, Antirez, and Aphyr is pretty long and I don’t really intend to dive into it in full (I probably don’t even know half of it tbh), but recently Jepsen did an audit of RedisRaft and published both a video and a post on it. I did find hilarious how much he was bashing Redis Labs in the video, and I think all of it is well deserved, but when it comes to bashing Redis itself, I can’t help but notice a hint of malice in the way everything was worded. Maybe it’s just me being too sensitive about something I care about.
As for the analogy, I think the bar for health inspectors is a bit higher: they certify a place is safe to eat, while given the nature of the tests Jepsen performs, it only states that the plates Aphyr ate did not contain bugs or hairs, nothing more than that. That’s the big premise of every Jepsen audit and you are basically told to draw your own conclusions, not too differently from Yelp.
Also, while technically correct to contextualize the audits as Jepsen does, it’s still kinda lame how the company can basically only deal slaps and nothing else.
But, as I said, maybe it’s just me being too sensitive :)
As for the analogy, I think the bar for health inspectors is a bit higher: they certify a place is safe to eat, while given the nature of the tests Jepsen performs, it only states that the plates Aphyr ate did not contain bugs or hairs, nothing more than that.
Well, all health inspectors do, ultimately, is certify that they didn’t find rats or unhealthy practices during their audit, right?
Regarding malice, I understand why you think the way you do. The distsys community kind of has their collective nose in the air about Redis, and it can manifest out in… not wonderful ways in backchannels and private conversations. But their frustrations aren’t totally unfounded. Redis is a fantastic tool with an extraordinary design and elegant implementation for what it is, which is a single-core single-machine data structure server. But each of the distsys features that Redis added were flawed, and not in an edge case or implementation detail sense but in a fundamentally unsound sense. The work reflected a kind of “let’s do this from first principals” approach to the problem space, which probably served the project very well for a lot of other features but is basically untenable when solving distsys problems. If a mis-step happens once it’s not a big deal, but Redis kept hammering away at different problems with this same attitude, and each of the solutions were similarly unsound. So the distsys folks kind of responded to that. Again, not well! But that’s the history there.
In any case, thanks for the context. I’m a happy user of, and advocate for, Redis, so it’s interesting to see your perspective.
Thank you, I’m also a happy user of go-kit and generally appreciative of your Go insights. I agree that Redis is far from perfect, I just think that learning to appreciate OSS projects in their full nature (flaws included) is the starting point to reclaiming control of OSS software from VC-funded madness. The technical debate should continue to exist, but lately I feel we’ve all been tricked into concentrating on that so that we wouldn’t notice what was happening on every other level.
given the nature of the tests Jepsen performs, it only states that the plates Aphyr ate did not contain bugs or hairs, nothing more than that. That’s the big premise of every Jepsen audit and you are basically told to draw your own conclusions, not too differently from Yelp.
I’m not happy about Hume’s problem of induction either, but it’s a real limitation of experimental inquiry, both theoretically and in practice. I can and do miss bugs, because modern distributed systems are incredibly complex, the space of faults under which they operate is intractably large, and I have limited time and make mistakes–not to mention that many of the core problems involved in this kind of work are, in general, NP-complete. The fact that experimental techniques work on concurrency verification at all is somewhat surprising, and has been the subject of several research papers.
I include these disclaimers in Jepsen because even though the limits of empiricism have been discussed for somewhere between 250 to 2000 years, users and vendors alike continue to cite Jepsen reports as “proof” of a distributed system’s correctness–and to treat Jepsen reports as if they were a uniform standard, rather than an investigatory process, tailored to the system in question, and shaped by the limits of time, money, computational power, space, system knowledge, and algorithmic know-how.
Acknowledging the limits of Jepsen’s methods is different from saying “draw your own conclusions”. In roughly six weeks of collaboration with Jepsen, Redis-Raft went from a system that could not execute a single operation or survive a single process fault, to providing strict serializability during hundreds of hours of mixed network and process faults. That’s nothing to sneeze at, and I noted as much in the talk and report.
Acknowledging the limits of Jepsen’s methods is different from saying “draw your own conclusions”.
As usual, you took a very long road to express in a ultimately self-celebratory way the same overall concept I sketched in two words. You’re also absolutely correct about the limits of the approach you employ and in your precision in pointing them out, as I also noted in my own, way less formal rant.
Unfortunately, that doesn’t make the end result any less lame.
I say this only because the author per se posted it to lobsters, so they seem to want randos like me to read it, but it definitely reads as “inside baseball”. I got pretty close to nothing from it except that the author is angry about a bunch of things that I haven’t heard about. Maybe some links would help provide context?
I just read the article, and I think it touches on bit more than just “inside baseball” (certainly aspects of inside baseball, but I don’t think it is only that).
I got 4 things from it.
Loris is leaving redis labs
Loris is joining the Zig software foundation
General rant (this definitely has an “inside baseball” aspect)
Loris is concerned/pissed/disappointed that big tech is strip mining communities.
While the “spontaneus software” world is busy infighting, big tech is systematically strip-mining it of any value, even when 99% of the value gets lost in the process. Tech companies are willing to go to any length to capture that 1%, and over time have learned to use Free software arguments against Free software itself, to ensure nobody would stop them.
The first two are communicative – a tech/comms person is leaving a project and joining another, and announcing it on a personal blog. Seems fine to me.
I think that last point is definitely worth fleshing out more, and I’d certainly like to see a future post with more of Loris’ thoughts on that.
Ask more precise questions and I’ll be happy to answer. I can’t cover everything in full detail in a single blog post, and the expectation that one single source is going to provide you with enough information to make up your mind is one of the reasons why marketing is so effective nowadays.
Not even sure the culture tag is the only one fitting.
I haven’t used Redis in a while and so I kinda missed the recent developments, including the described “every company wants it to do something different” but sounds perfectly reasonable for a project that I still remember as “nicely written small replacement for memcache, oh it and has a queue, etc.pp” :)
The only thing I didn’t understand was the dig at Jepsen, there must be another story I have missed,
This is a really strange take on the role that Jepsen plays in the ecosystem. Kyle isn’t motivated by any of the factors that make Yelp so pathological; if we’re going to place Jepsen into the restaurant metaphor, I’d say they’re the health inspector. Can you say more?
Hi Peter, it’s part of the truly ranty part of the post (and explicitly marked as such), so I wouldn’t take it too seriously, but in this one blog post dedicated to describing my experience with Redis, I wanted to call out all the organizations that in one way or another I feel have not done justice to Redis, even if not directly and without monetary reasons to do so.
The history between Redis, Antirez, and Aphyr is pretty long and I don’t really intend to dive into it in full (I probably don’t even know half of it tbh), but recently Jepsen did an audit of RedisRaft and published both a video and a post on it. I did find hilarious how much he was bashing Redis Labs in the video, and I think all of it is well deserved, but when it comes to bashing Redis itself, I can’t help but notice a hint of malice in the way everything was worded. Maybe it’s just me being too sensitive about something I care about.
As for the analogy, I think the bar for health inspectors is a bit higher: they certify a place is safe to eat, while given the nature of the tests Jepsen performs, it only states that the plates Aphyr ate did not contain bugs or hairs, nothing more than that. That’s the big premise of every Jepsen audit and you are basically told to draw your own conclusions, not too differently from Yelp.
Also, while technically correct to contextualize the audits as Jepsen does, it’s still kinda lame how the company can basically only deal slaps and nothing else.
But, as I said, maybe it’s just me being too sensitive :)
Well, all health inspectors do, ultimately, is certify that they didn’t find rats or unhealthy practices during their audit, right?
Regarding malice, I understand why you think the way you do. The distsys community kind of has their collective nose in the air about Redis, and it can manifest out in… not wonderful ways in backchannels and private conversations. But their frustrations aren’t totally unfounded. Redis is a fantastic tool with an extraordinary design and elegant implementation for what it is, which is a single-core single-machine data structure server. But each of the distsys features that Redis added were flawed, and not in an edge case or implementation detail sense but in a fundamentally unsound sense. The work reflected a kind of “let’s do this from first principals” approach to the problem space, which probably served the project very well for a lot of other features but is basically untenable when solving distsys problems. If a mis-step happens once it’s not a big deal, but Redis kept hammering away at different problems with this same attitude, and each of the solutions were similarly unsound. So the distsys folks kind of responded to that. Again, not well! But that’s the history there.
In any case, thanks for the context. I’m a happy user of, and advocate for, Redis, so it’s interesting to see your perspective.
Thank you, I’m also a happy user of go-kit and generally appreciative of your Go insights. I agree that Redis is far from perfect, I just think that learning to appreciate OSS projects in their full nature (flaws included) is the starting point to reclaiming control of OSS software from VC-funded madness. The technical debate should continue to exist, but lately I feel we’ve all been tricked into concentrating on that so that we wouldn’t notice what was happening on every other level.
I’m not happy about Hume’s problem of induction either, but it’s a real limitation of experimental inquiry, both theoretically and in practice. I can and do miss bugs, because modern distributed systems are incredibly complex, the space of faults under which they operate is intractably large, and I have limited time and make mistakes–not to mention that many of the core problems involved in this kind of work are, in general, NP-complete. The fact that experimental techniques work on concurrency verification at all is somewhat surprising, and has been the subject of several research papers.
I include these disclaimers in Jepsen because even though the limits of empiricism have been discussed for somewhere between 250 to 2000 years, users and vendors alike continue to cite Jepsen reports as “proof” of a distributed system’s correctness–and to treat Jepsen reports as if they were a uniform standard, rather than an investigatory process, tailored to the system in question, and shaped by the limits of time, money, computational power, space, system knowledge, and algorithmic know-how.
Acknowledging the limits of Jepsen’s methods is different from saying “draw your own conclusions”. In roughly six weeks of collaboration with Jepsen, Redis-Raft went from a system that could not execute a single operation or survive a single process fault, to providing strict serializability during hundreds of hours of mixed network and process faults. That’s nothing to sneeze at, and I noted as much in the talk and report.
As usual, you took a very long road to express in a ultimately self-celebratory way the same overall concept I sketched in two words. You’re also absolutely correct about the limits of the approach you employ and in your precision in pointing them out, as I also noted in my own, way less formal rant.
Unfortunately, that doesn’t make the end result any less lame.
I say this only because the author per se posted it to lobsters, so they seem to want randos like me to read it, but it definitely reads as “inside baseball”. I got pretty close to nothing from it except that the author is angry about a bunch of things that I haven’t heard about. Maybe some links would help provide context?
I just read the article, and I think it touches on bit more than just “inside baseball” (certainly aspects of inside baseball, but I don’t think it is only that).
I got 4 things from it.
The first two are communicative – a tech/comms person is leaving a project and joining another, and announcing it on a personal blog. Seems fine to me.
I think that last point is definitely worth fleshing out more, and I’d certainly like to see a future post with more of Loris’ thoughts on that.
Ask more precise questions and I’ll be happy to answer. I can’t cover everything in full detail in a single blog post, and the expectation that one single source is going to provide you with enough information to make up your mind is one of the reasons why marketing is so effective nowadays.
Not even sure the culture tag is the only one fitting.
I haven’t used Redis in a while and so I kinda missed the recent developments, including the described “every company wants it to do something different” but sounds perfectly reasonable for a project that I still remember as “nicely written small replacement for memcache, oh it and has a queue, etc.pp” :)
The only thing I didn’t understand was the dig at Jepsen, there must be another story I have missed,
I suggested the
ranttag since there’s no technical content here and the tone is somewhat hyperbolic and inflammatory.I actually meant the other direction towards technical but I guess you’re not wrong either :)