I realize it wasn’t the purpose of the article, but curious to see this on non-Linux OSes (e.g. the BSDs, Darwin).
I’ve been watching Seagate’s Kinetic Platform, which is basically a reworking of a disk drive to directly accept an ethernet connection and expose the disk’s native storage system as a more closely-aligned key-value store API. It tries to remove metadata/filesystem handling latency from the I/O.
Among the cool things this system allows: direct transfer of key-values from one drive to another.
It is highly likely that I don’t know what this should be used for, but I think some of the limits might be a bit, er, limiting:
The key/value entries are limited in size to 4-KiB keys and 1-MiB values.”
ie. You can’t store a photo or a music file in a single value.
That’s not a problem. Many key/value stores limit the maximum size of values and depend on a layer above it to break down larger files into the 1MB chunks. FoundationDB, for example, has a “blob” layer that does exactly this. Also, the layer above it would want to do some kind of replication or erasure coding in case a drive fails. The fault tolerance is better if you break large files into small pieces and distribute them across a number of drives.
OSv is a new kernel written in C++ with Linux compatibility. They claim 2x throughput for unmodified Redis.
Yes, but the point is that we want to lower latency, not throughput.
I think 2x throughput will also mean 0.5x latency in this case.
Oof. Looks like that OS doesn’t support users, which might be at least moderately reasonable. But it might not support processes, which might make certain servers difficult to handle. Maybe it silently translates all forks into thread spawns?
Anyway, interesting idea, but there are some tradeoffs I’m not yet comfortable with.
I recommend viewing the Arrakis OSDI presentation for more technical details. (It won best paper at OSDI 14) Also worth checking out is the Barrelfish “Multikernel” that it is derived from: site, paper.