Have you considered Posix MQ? We might call it „serious MQ“ and you probably already have it in your operating system. No additional broker or server needed.
BTW: Posix MQ (and other messaging systems) will be supported in the next release of Relational pipes. It integrates well with shell scripts. Some early implementation is already here: https://hg.globalcode.info/relpipe Any tips are welcomed (what would you expect from such integration etc.).
happy you liked it, indeed the article was written more to explain concepts like that using simple components rather than, idk, meeting the demand of shell-based message queues (lmao)
A bit of a tangent: when, and why, did people starting refering to these things as message queues rather than queues? A queue is always a message queue.
one disadvantage of pipes is that they’re not restartable: if the consumer fails the message is lost. that could be bad.
another disadvantage (related to the first) is that the consumer has to run concurrently with the producer. that might make it harder to manage resources well enough to prevent consumer-failure
another disadvantage of pipes is that they’re only atomic up to PIPE_BUF.
another disadvantage of pipes is that in order to have more consumers than producers, the producer needs to do routing. which logfiles, the consumers can simply skip every N-1 records (where N is the size of the group).
if these disadvantages are important, you probably should not use a pipe, but if not, there’s nothing preventing your from also using a fifo as the source for your logs (you can use cat if you want).
i don’t recommend the approach in the article for crashproof consumption, or the message format: the message should begin with a newline (not end in one) to protect against losing the first good message (in addition to the last bad one you can do nothing about at this point). there’s too much copying there. instead there is a much simpler approach: record the position in the logfile as you make progress and restart from that position if you are (well) restarted. you can even rotate logfiles to create the illusion of one really big logfile that is perhaps easier to manage for the sysadmin, just remember to consider the filename to be part of the position.
Have you considered Posix MQ? We might call it „serious MQ“ and you probably already have it in your operating system. No additional broker or server needed.
BTW: Posix MQ (and other messaging systems) will be supported in the next release of Relational pipes. It integrates well with shell scripts. Some early implementation is already here: https://hg.globalcode.info/relpipe Any tips are welcomed (what would you expect from such integration etc.).
I love the simple explanation of a WAL. Might be the best part of the article, imho.
happy you liked it, indeed the article was written more to explain concepts like that using simple components rather than, idk, meeting the demand of shell-based message queues (lmao)
It’s been a month or so and I’m still bringing up this article in conversations at work.
That’s cool , how so?
Honestly, it’s the way the post builds and progresses. The story it tells is outstanding.
A bit of a tangent: when, and why, did people starting refering to these things as message queues rather than queues? A queue is always a message queue.
Not when you are in queue for a bank teller.
wouldn’t you just use a pipe
one disadvantage of pipes is that they’re not restartable: if the consumer fails the message is lost. that could be bad.
another disadvantage (related to the first) is that the consumer has to run concurrently with the producer. that might make it harder to manage resources well enough to prevent consumer-failure
another disadvantage of pipes is that they’re only atomic up to
PIPE_BUF.another disadvantage of pipes is that in order to have more consumers than producers, the producer needs to do routing. which logfiles, the consumers can simply skip every N-1 records (where N is the size of the group).
if these disadvantages are important, you probably should not use a pipe, but if not, there’s nothing preventing your from also using a fifo as the source for your logs (you can use cat if you want).
i don’t recommend the approach in the article for crashproof consumption, or the message format: the message should begin with a newline (not end in one) to protect against losing the first good message (in addition to the last bad one you can do nothing about at this point). there’s too much copying there. instead there is a much simpler approach: record the position in the logfile as you make progress and restart from that position if you are (well) restarted. you can even rotate logfiles to create the illusion of one really big logfile that is perhaps easier to manage for the sysadmin, just remember to consider the filename to be part of the position.
Thank you! So much great insight in this comment!