One nice feature of FIFO’s is that if you write less than PIPE_BUF bytes, the write is atomic. This is not necessarily so useful in the 1 producer multiple consumer case, but in multiple producers one consumer case it means the file lock can be removed alltogether. Unfortunately, PIPE_BUF is small on OS’s like FreeBSD (512 bytes, I think), but Linux is around 8kb, I think.
The other thing to note is that the FIFO is not doing fan-out. Each consumer is destructive to the FIFO.
All-in-all, I think FIFOs are an underused tool in a lot of software. The kernel is pretty efficient at shuffling data around, the cost is a a syscall to read/write to a FIFO, but it’s a nice way to connect components and maintain process boundaries. Unless someone is doing really high throughput/low latency work where disrupting the CPU cache is a serious problem, using FIFOs to connect components will probably have no negative effect on performance.
Thank you for your comment. FIFOs can be a bit limited in terms of atomic writes, and in they are indeed not recommended for multiple consumers. I’ve extended the post to add more details about some of the limitations.
@wsdookadr since you made a blog post, a few comments about how it works would be nice for the non bash/*nix trivial stuff. E.g. how it works in general, what is ‘200>$LOCKFILE’, what does flock do, etc.
One nice feature of FIFO’s is that if you write less than
PIPE_BUFbytes, the write is atomic. This is not necessarily so useful in the 1 producer multiple consumer case, but in multiple producers one consumer case it means the file lock can be removed alltogether. Unfortunately,PIPE_BUFis small on OS’s like FreeBSD (512 bytes, I think), but Linux is around 8kb, I think.The other thing to note is that the FIFO is not doing fan-out. Each consumer is destructive to the FIFO.
All-in-all, I think FIFOs are an underused tool in a lot of software. The kernel is pretty efficient at shuffling data around, the cost is a a syscall to read/write to a FIFO, but it’s a nice way to connect components and maintain process boundaries. Unless someone is doing really high throughput/low latency work where disrupting the CPU cache is a serious problem, using FIFOs to connect components will probably have no negative effect on performance.
Thank you for your comment. FIFOs can be a bit limited in terms of atomic writes, and in they are indeed not recommended for multiple consumers. I’ve extended the post to add more details about some of the limitations.
@wsdookadr since you made a blog post, a few comments about how it works would be nice for the non bash/*nix trivial stuff. E.g. how it works in general, what is ‘200>$LOCKFILE’, what does flock do, etc.