Zk has a far better track record of reliability than hashicorp components. Seems like a pretty clear net loss IMO, but if you hate dealing with a JVM that badly then maybe it makes you feel better.
Can you elaborate on the reliability problems? I’m not deeply plugged in (other than reading Aphyr’s posts whenever possible) but I have admired the hashicorp components from afar for a long time.
It seems like working towards making the state-keeping implementation in Kafka pluggable to use other strongly consistent backing stores would be a better use of effort than rewriting Kafka.
I didn’t find the dependency on ZK to be too bad when I was playing around with Kafka. ZK is interesting technology in its own right. What did annoy me was that the book I Heart Logs advocated a way of using Kafka that the official Kafka software doesn’t support all that well. Additionally, I believe they do store in ZK the last offset you’ve read on the stream, which just seems like a bad idea. You can ignore that though and tell it where to start.
I’ve since learned about Kafka Streams which is closer to the approach outlined in I Heart Logs.
I like Kafka, but I wound up using RabbitMQ on this project. Kafka is kind of a “leaky abstraction”; I imagine most people using it don’t care because they need the distribution and raw performance, where it still has a significant (~10x, IIRC) advantage over the open source AMQP implementations. I found I liked AMQPs abstractions a lot better too. In a few years, I will look at Kafka again.