Huh. If it’s fine to create your own “mediation pipe” to satisfy the API, and then splice(in, pipe); splice(pipe, out) … why doesn’t the kernel support that itself?
I was working with an embedded system where, for installation/testing purposes (details forgotten), we had a service running in a VM that would help bootstrap a system. Part of this setup was a script that would run on the target, read a binary blob from a file descriptor and write it to the device. That reading was done with the target system’s implementation of cat.
As it happened, the sending of the binary blob to the target turned out to be a significant bottleneck. After I profiled it, it turned out that cat was taking a long time to read and write the bytes. I looked at the source and found it was using fread and fwrite. I changed it to use read/write and the transfer time went down significantly. It was great because I’m fairly certain this was part of our automated build system to create images for the devs and the result was that build times went down a lot.
With read/write, it’s just copying into a buffer (read), and then copying out of that buffer (write).
With fread/fwrite it’s copying into a buffer (read), then copying from that buffer to another buffer (fread), then potentially copying that back (fwrite), before finally copying it back into another buffer (write). That can really add up - even the read/write loop is more than we’d want ideally, hence stuff like splice.
I’d imagine that the reason for GNU cat not using splice is because of portability (POSIX). Remember GNU cat wants to support different OS’es. Splice being a Linux specific feature would break this portability.
Another surprise: Rust’s
std::fs::copydoes in-kernel copy when possible, usingcopy_file_rangesyscall on Linux.Huh. If it’s fine to create your own “mediation pipe” to satisfy the API, and then
splice(in, pipe); splice(pipe, out)… why doesn’t the kernel support that itself?Maybe it’s because errors need to be reported on the
pipeinstead of oninorout?I wrote a fast
catonce out of actual necessity.I was working with an embedded system where, for installation/testing purposes (details forgotten), we had a service running in a VM that would help bootstrap a system. Part of this setup was a script that would run on the target, read a binary blob from a file descriptor and write it to the device. That reading was done with the target system’s implementation of
cat.As it happened, the sending of the binary blob to the target turned out to be a significant bottleneck. After I profiled it, it turned out that
catwas taking a long time to read and write the bytes. I looked at the source and found it was usingfreadandfwrite. I changed it to useread/writeand the transfer time went down significantly. It was great because I’m fairly certain this was part of our automated build system to create images for the devs and the result was that build times went down a lot.So sometimes you really do need a faster
cat.Why were fread/fwrite slow? Aren’t they thin veneers over read/write that do a bit of buffering?
For “buffering”, read “copying”.
With read/write, it’s just copying into a buffer (
read), and then copying out of that buffer (write).With fread/fwrite it’s copying into a buffer (
read), then copying from that buffer to another buffer (fread), then potentially copying that back (fwrite), before finally copying it back into another buffer (write). That can really add up - even the read/write loop is more than we’d want ideally, hence stuff likesplice.Does it work properly with stdin and EOFs sent with ^D?
Yes.
I didn’t know this was an issue. :-)
tail -f -hitting ctrl+d does not quit the program, butcat -does.I assume using splice from Ruby would result in a similar increase in speed.
I’d imagine that the reason for GNU
catnot using splice is because of portability (POSIX). Remember GNUcatwants to support different OS’es. Splice being a Linux specific feature would break this portability.Not really. GNU
cptakes advantage of Linux-onlyFICLONE(copy-on-write file cloning) feature when available. GNUcatcould do the same.