Introducing Faktory - a new background job system

26

Introducing Faktory - a new background job system ☶ show programming mikeperham.com
authored by mperham 5 years ago | archive
Archive.org Archive.today Ghostarchive
| 13 comments

13

Been working on this quietly for the last six months. I hope people find this useful!

1. 3
  
  bogdan edited 5 years ago | link
  
  This is cool, congrats! I’m going to play around with using it as a broker in dramatiq this weekend :D. A couple questions:
  
  From what I can tell, a lot of things Faktory does overlap with things RabbitMQ provides ootb. How are you planning to differentiate long term?
  
  Is the protocol documented anywhere? I haven’t been able to find anything. The best I could find was this file in the Ruby client.
  1. 1
    
    mperham 5 years ago | link
    
    The wiki has some protocol docs. Right now, the wiki + faktory_worker_{go,ruby} are the things to look into.
2. 1
  
  mempko 5 years ago | link
  
  So it’s like a tuple space with a JSON job definition and a nice UI. That’s pretty cool.
3. 1
  
  sovanesyan 5 years ago | link
  
  I couldn’t find information on how you plan to handle job persistence. What happens to the jobs if the server fails?
  1. 3
    
    mperham 5 years ago | link
    
    TBD. This is the public unveiling of a new project. No one ships with a perfect HA solution on day one.
    1. 2
      
      sovanesyan edited 5 years ago | link
      
      I am not looking for a perfect solution here, just wondering what you are planning for the project.
      
      An embedded database implies you want to build the HA logic in the application layer, as opposed to re-use the database capabilities. Is that the case?
      
      Note: I am in no way familiar with RocksDB, maybe there is something very obvious I don’t understand.
      1. 2
        
        mperham 5 years ago | link
        
        “TBD” sort of covers my plans right now. I’ve heard RocksDB has something internal which can be massaged into a replication stream but I’m not deep enough to grok it yet.
        
        1
        
        sovanesyan 5 years ago | link
        
        Okay, I’ll look into it as well, thanks.
  2. 1
    
    mulander 5 years ago | link
    
    from the article itself:
    
    Faktory goes further and provides the same job persistence, state management and monitoring Web UI that Sidekiq does. It uses Facebook’s high-performance RocksDB embedded datastore internally to persist all job data, queues, error state, etc.
    1. 2
      
      ac 5 years ago | link
      
      I think he meant if the rocksdb server goes down, implying it has no fault tolerance.
      1. 3
        
        inactive-user 5 years ago | link
        
        It’s no worse a problem than relying on Postgres, minus the fact that RocksDB doesn’t have an obvious replication strategy.
        
        So, to recap: fine for a hobby app, requires more effort for anything critcal.
      2. 1
        
        skade 5 years ago | link
        
        rocksdb has no server, it’s an embedded database. It’s a fork of LevelDB.
        
        1
        
        sovanesyan 5 years ago | link
        
        Exactly, we were talking about the server hosting Faktory and in turn RocksDb.