1. 14
  1.  

  2. 1

    Dramatiq is licensed under the AGPL

    Now I have three options:

    • Make the codebase at work opensource (lol)
    • Violate AGPL on purpose
    • Use any of the other Redis-based task queues

    I really don’t get what the author is trying to achieve with choosing a license like this.

    1. 13

      You could also buy a license.

      1. 1

        Ah, that makes sense. I didn’t see that, shame on me.

      2. 4

        I think commercial backing of some sort or another is the only way we can sustainably develop open source software long term and dual licensing seemed like the lowest friction way to get started. I’ll have to highlight that fact a little better in the docs! :D

        1. 2

          You are right of course. Other message frameworks like sidekiq seem to do alright: https://github.com/mperham/sidekiq

          The challenge here is that Celery is in pretty great shape for a free solution. On the other hand Python’s support for high concurrency is changing rapidly so who knows maybe there’s room for a new player in this market.

          1. 2

            I’ve never met anyone IRL who’s worked with Celery and didn’t run into problems, so there’s definitely room for improvement in this area.

            1. 2

              It works like a charm with RabbitMQ as a backend. The rest is pretty experimental and breaks, especially at high volume. (I’ve been using Celery for >5 years)

              1. 4

                I’ve been using Celery professionally for about 3 years and dramatiq tries to solve many of the issues I’ve encountered using it. Some stuff that immediately springs to mind:

                • Celery doesn’t support task prioritization. You have to deploy multiple sets of workers in order to prioritize queues.
                • Celery has poor support for delayed tasks. Delayed tasks go on the same queue that normal tasks go on and they’re simply pulled into worker memory until they can be executed. This makes it hard to autoscale workers by queue size.
                • Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.
                • Celery tasks are not retried on error by default.
                • Celery’s not well suited for integration testing. You’re expected to unit test tasks and to turn eager evaluation on for integration tests, but even then task exceptions will be swallowed by default. Dramatiq provides an in-memory stub broker specifically for this use case.
                • The source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.
                • It’s easy for some of its more advanced “canvas” features to drop tasks.

                All of the above are things that are first-class in dramatiq and there are definitely other things I’m not thinking of right now. That’s not to say that celery is bad, but I think we can do better and that’s why I made dramatiq. :D

                1. 1

                  Considering your experience, I was wondering what’s your take on rq? (others who used it, are obviously welcomed to chime in too)

                  1. 1

                    I don’t have much experience with RQ since it is Redis-only and I’ve generally preferred to use RabbitMQ as a message broker. However, a few things that seem like disadvantages to me with RQ are:

                    • Messages are pickled so it’s strictly limited to Python and pickled messages are potentially exploitable. This also means you may sometimes send bigger messages than you intended over the network purely by accident.
                    • Queue prioritisation is handled like it is in Celery: you have to spawn different sets of workers.
                    • It forks for every job, so it’s slightly slower and forks that are killed b/c they’ve surpassed their time limits can leak DB connections if you’re not careful. I understand this may be swappable behaviour, however.
                    • Similar to Celery, there isn’t a good integration testing story for RQ.

                    Because I’ve criticised both Celery and RQ at this point, I feel it’s important that I mention a couple areas where they’re both currently better than dramatiq:

                    • the obvious one: it’s newer than either of those and is less likely to be familiar to users. The extension ecosystem for dramatiq is nonexistent (though I will be releasing integration packages for Django and Flask soon!)
                    • dramatiq doesn’t store task results and doesn’t offer a way to retrieve them. Adding that sort of functionality is trivial using middleware, but it’s not there ootb so if you absolutely need something like that and you don’t care about the things I have mentioned so far then you should look at Celery or RQ instead.
                    1. 1

                      Thank you for taking the time to post this!

                      There are two other areas that bother me personally:

                      • Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.
                      • The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.

                      I share @jscn’s sentiment about Celery. I I was wondering if RQ, despite the above disadvantages might be more stable. At least their codebase should easier to grok (single repo)…

                      1. 1

                        Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.

                        I’m considering adding Python 2 support, but it’s a hard thing to balance what with 2.x getting EOL’d in a little less than 2 and a half years.

                        The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.

                        Understandable.

                  2. 1

                    Sure, that’s true. Did you ever look at https://github.com/RichardKnop/machinery that project is still really early. Probably much easier to compete with.

            2. 1

              beanstalkd, NSQ, resque, celery, huey, … — pretty much everything in this space is non-GPL. So “use any other queue thing” will definitely be a very popular option :)

              1. 5

                So “use any other queue thing” will definitely be a very popular option :)

                That’s perfectly fine! I just want those people that get value out of my work to contribute back in some way. If someone makes a cost-benefit analysis and decides that they’d rather use celery over dramatiq because they prefer the cheaper option (although it’s worth mentioning that I give out free comm. licenses for one year per company) then that’s their prerogative. I’ll still be around a year later when they realise their mistake ;).

            3. 2

              Trying to achieve you not using this at work? That’s usually what I’m going for when I choose AGPL

            4. 1

              If I understand the docs, it seems like it’s on me, the user, to deploy and manage the code the workers run. It would be great (and a differentiating feature) if dramatiq could handle this – e.g., if the workers were dumb and ran code distributed by the submitter. Deploying code to dozens or hundreds of workers is always what makes these things painful to use.

              1. 1

                IIRC Cloud Haskell works in the way you propose. While interesting, I don’t think it’s a model I’d adopt for a dynamic language. At least not for your average SaaS workload!

              Stories with similar links:

              1. Dramatiq – An alternative to Celery authored by bogdan 8 months ago | 12 points | 1 comment