1. 7

  2. 4

    Two quick thoughts:

    1. There are companies that manage Kafka for you, such as Heroku. One of the points this article makes is that there’s a time/energy cost to your org’s engineers spent on managing your own Kafka cluster, but that doesn’t have to be true.
    2. The post alludes to but skirts around the cost of vendor lock-in. If you want to move away from Amazon and you are using an open source queueing system / event stream, it requires less ops & engineering time than having to change from a proprietary system to a new less-well-understood system in the midst of the rest of the migration process.
    1. 2

      I had a lot of stability trouble using the Java Kinesis Client Library in a high-throughput system - though technically this was for DynamoDB Stream which uses an adapter for the KCL, but is subtly different from Kinesis itself I gather. That may have been an exacerbating factor since there’s less control over sharding there.

      It does look like they released a fix for a cause of KCL instability, but it took them almost a year to release the fix despite having identified the problem the same day it was reported. That was disappointing, and ultimately too late for what we were evaluating it for.

      Seeing this article reminded me to check up on the KCL issue, so I’m happy they plugged that hole and it’s seen some improvements since the last time I checked.