1. 26

  2. 5

    we’re able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack

    We designed a new type of mbuf that could carry up to 24 pages for sendfile, and which could also carry the TLS header and trailing information in-line

    That’s… not 100% default anymore :D But it’s kinda cool that they’re hacking the kernel instead of reaching for netmap. (netmap was created for a reason though, I guess…)

    1. 2

      It looks like freebsd’s mbuf API allows you to customize it quite a bit, so in that sense, if I’m understanding the post correctly, they didn’t have to modify the TCP stack to get the flexibility they needed to accomplish this.

      1. 4

        No, customization is modification, and they did quite a lot of modification:

        We had to augment quite a few functions in the network stack to use accessor functions to access the mbufs, teach the DMA mapping system (busdma) about our new mbuf type, and write several accessors for copying mbufs into uios, etc. We also had to examine every NIC driver in use at Netflix and verify that they were using busdma for DMA mappings, and not accessing parts of mbufs using mtod().

        1. 2

          it depends. if they got all of these changes landed in the stock kernel, which doesn’t seem impossible, one could still call it stock. I read this as partially tooting their own horn for increasing the scalability of the freebsd TCP stack.