1. 26

Been working on this quietly for the last six months. I hope people find this useful!

  1.  

  2. 3

    This is cool, congrats! I’m going to play around with using it as a broker in dramatiq this weekend :D. A couple questions:

    From what I can tell, a lot of things Faktory does overlap with things RabbitMQ provides ootb. How are you planning to differentiate long term?

    Is the protocol documented anywhere? I haven’t been able to find anything. The best I could find was this file in the Ruby client.

    1. 1

      The wiki has some protocol docs. Right now, the wiki + faktory_worker_{go,ruby} are the things to look into.

    2. 1

      So it’s like a tuple space with a JSON job definition and a nice UI. That’s pretty cool.

      1. 1

        I couldn’t find information on how you plan to handle job persistence. What happens to the jobs if the server fails?

        1. 3

          TBD. This is the public unveiling of a new project. No one ships with a perfect HA solution on day one.

          1. 2

            I am not looking for a perfect solution here, just wondering what you are planning for the project.

            An embedded database implies you want to build the HA logic in the application layer, as opposed to re-use the database capabilities. Is that the case?

            Note: I am in no way familiar with RocksDB, maybe there is something very obvious I don’t understand.

            1. 2

              “TBD” sort of covers my plans right now. I’ve heard RocksDB has something internal which can be massaged into a replication stream but I’m not deep enough to grok it yet.

              1. 1

                Okay, I’ll look into it as well, thanks.

          2. 1

            from the article itself:

            Faktory goes further and provides the same job persistence, state management and monitoring Web UI that Sidekiq does. It uses Facebook’s high-performance RocksDB embedded datastore internally to persist all job data, queues, error state, etc.

            1. 2

              I think he meant if the rocksdb server goes down, implying it has no fault tolerance.

              1. 3

                It’s no worse a problem than relying on Postgres, minus the fact that RocksDB doesn’t have an obvious replication strategy.

                So, to recap: fine for a hobby app, requires more effort for anything critcal.

                1. 1

                  rocksdb has no server, it’s an embedded database. It’s a fork of LevelDB.

                  1. 1

                    Exactly, we were talking about the server hosting Faktory and in turn RocksDb.