Thanks, it’s a great story. LZ4 really was a game changer (ZSTD is nicer). It was so fast that even a slow CPU wasn’t a bottleneck, so unless you really cared about latency enabling it probably made things faster. I remember it becoming the default for ZFS and it really was a huge win: things got a bit smaller, but they got faster by the same amount. The total amount of data read or written was reduced and so the disk became less of a bottleneck but, unlike gzip mode, it wasn’t replaced by a bottleneck on the CPU.
As he mentions, doing less was important, but the big win for LZ4 was the early abort. If you’re writing a zip file or an MPEG stream to disk, LZ4 bails early. It isn’t going to get a good compression ratio on already-compressed (or encrypted) data, so it detects that and bails. In contrast, gzip mode just sat there eating your CPU (and often made thing bigger!).
It’s also great to read that even people like Yann suffer from imposter syndrome at times!
Tangentially, I remember seeing it mentioned in some benchmark of LZO that sometimes LZO decompression could occasionally be a little faster than memcpy because it needed to read fewer bytes from memory.
Yann Collet’s story is crazy I really loved the interview, thank you.
One thing that resonates with me a lot: to stay motivated in the long run (years and years) on a dedicated and hard subject, your driver should be deep inside you. It’s not the money that will keep you digging into the little details, gaining a small perf percentage at each step.
Host here. Yann Collet was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator.
Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders.
His code ended up everywhere: in games, databases, file systems, and the Linux Kernel because Yann built the world’s fastest compression algorithms.
And he got started just making a fun game for a graphing calculator he’d had since high school.
Thanks, it’s a great story. LZ4 really was a game changer (ZSTD is nicer). It was so fast that even a slow CPU wasn’t a bottleneck, so unless you really cared about latency enabling it probably made things faster. I remember it becoming the default for ZFS and it really was a huge win: things got a bit smaller, but they got faster by the same amount. The total amount of data read or written was reduced and so the disk became less of a bottleneck but, unlike gzip mode, it wasn’t replaced by a bottleneck on the CPU.
As he mentions, doing less was important, but the big win for LZ4 was the early abort. If you’re writing a zip file or an MPEG stream to disk, LZ4 bails early. It isn’t going to get a good compression ratio on already-compressed (or encrypted) data, so it detects that and bails. In contrast, gzip mode just sat there eating your CPU (and often made thing bigger!).
It’s also great to read that even people like Yann suffer from imposter syndrome at times!
For sure! Going to work as a professional developer for the first time in your 40s at Facebook sounds very intimidating.
I didn’t even know about that. Very cool!
Tangentially, I remember seeing it mentioned in some benchmark of LZO that sometimes LZO decompression could occasionally be a little faster than memcpy because it needed to read fewer bytes from memory.
Yann Collet’s story is crazy I really loved the interview, thank you.
One thing that resonates with me a lot: to stay motivated in the long run (years and years) on a dedicated and hard subject, your driver should be deep inside you. It’s not the money that will keep you digging into the little details, gaining a small perf percentage at each step.
Agreed, that was a big take away of mine as well. You need to be in it for the long term and have deep motivations for that.