would be interested to hear experiences with Pulsar.
At one point in time, I was looking for a pub/sub to send log messages from end-user handling web-backends, to middle office apps.
The use-case, of course, too simple for a true pub/sub.
But I also wanted to use a technology that once we adapt, could be used for other more ‘demanding scenarios’.
So I was looking at Pulsar and Aeron ( https://github.com/real-logic/Aeron )
I liked aeron easiness of deployment.
At the end, we just configured Java logging to use a centralized rsyslog.
It solved our immediate need so that our investigation did not go anywhere…
Why would I use this instead of kafka? Presumably this either has a different sweet spot or is an attempt to be better in a way that is concretely articulable?
Other than the ecosystem (which shouldn’t be discounted), Pulsar is quite a bit of ahead in terms of architecture and API decisions than Kafka. Arguably, they had the second mover advantage.
One key feature I particularly like is that storage is offloaded to Bookkeeper rather than storing it locally on broker nodes. Thus storage is decoupled from the brokers (where as Kafka stores data on broker nodes). This makes scaling the storage and compute (brokers) trivially easy. The trade-off is that you need to operate a Bookkeeper cluster (in addition to the brokers and Zookeeper).
In addition, storage can be “tiered” which means, old messages (by some definition of old) can be offloaded transparently to object storage such as S3. This is often cheaper storage than what Bookkeeper uses at the expense of latency when requesting it. It may not be necessary, but its a nice option for keeping all the data and archiving it.
Personally, I am keeping an eye on the NATS project, specifically the new development around Jetstream (https://github.com/nats-io/jetstream#readme). The simplicity of the API and operability of the cluster (no external dependencies) is very attractive. As the README notes, its still in development, but the creator Derek Collison is actively working on clustering and remaining features over the next few months.
Don’t know if it is ok to link to the angry orange site, but here’s a number of comments directly addressing your question: