I never understood job queues that aren’t transactional with your main database. It seems like asking for subtle timing bugs. I currently use a job queue I wrote for go that installs the jobs in postgres in the same transaction other job dependencies appear.
For example If I want to create a user row in my database, and pend the creation of an external stripe customer account, I make sure that happens in the same transaction. It seems like with queues like this you can end up with customers with missing stripe accounts with some unlucky shutdown timing. Then wouldn’t you just need lots of ugly check and retry logic (Basically business specific fsck)?
Is this something other people have considered? or am I over reacting?
Depends on your use case. I’m hacking on something where a job queue is critical but there actually is no database or persistent state at all.
That’s one of those cases where maybe queues aren’t the best idea. I find them useful for handing tasks that can take seconds to minutes to complete in response to front end activity but that do not require immediate user interaction. e.g processing of uploaded images; once an image is uploaded the backend can make it available to the frontend instantly while a spawned task works on cropping/optimising/etc.
I won’t believe your interface is generic until you have at least 2 implementations ;-p
How does this compare to for example Machinery https://github.com/RichardKnop/machinery