1. 35

  2. 8

    Good article, which I would suggest to everyone who wants to start to use message broker.

    For short-lived tasks, publish-subscribe is a convenient way to build a system quickly, but you inevitably end up implementing a new protocol atop. You have publish-subscribe, but you really want request-response. If you want something computed, you’ll probably want to know the result.

    Starting with publish-subscribe makes work assignment easy: jobs get added to the queue, workers take turns to remove them. Unfortunately, it makes finding out what happened quite hard, and you’ll need to add another queue to send a result back.

    Once you can handle success, it is time to handle the errors. The first step is often adding code to retry the request a few times. After you DDoS your system, you put a call to sleep(). After you slowly DDoS your system, each retry waits twice as long as the previous.

    This happened to me in my last job. One thing I learned was that I don’t really want to reinvent RPC framework and all the fluff around it myself.

    1. 11

      This is a great post with lots of insights into the tradeoffs of different ways to split up a system. When the conclusion is basically “it depends” you know it’s a great source of information and not a marketing piece cloaked as engineering advice.

      1. 3

        Worth reading, but I’d like to see a counterpoint article, since there are a few points in this article that could be debated (e.g., the staggered-retry commentary).