A while ago I wrote a simple ttftrim tool to remove glyphs, which is a bit easier to use than that Font Subsetter tool as you can just do:
$ ttftrim font.ttf sub.ttf ~/path/to/my/site
And it will scan all the files and include only the used glyphs.
This could be a bit smarter still, for example the only reason I have Ø in my font is because it’s used in a codeblock (which uses a monospace font), but you can remove them manually if you want. It also doesn’t modify lookup tables so you can’t remove some stuff that’s referenced there (I couldn’t figure out how to do that with the Fontforge library), but it already reduced the total size on my site from 144K to 55K, so not a bad win nonetheless.
Didn’t know about font-display – thank you!