I think the goal with federated systems should be that every single person has their own home server. It can be a cheap computer, like a Raspberry Pi, that is always available and stores your data in one place. You can then access the service with your PC, Laptop, Smartphone, etc. Everything is under your control and you get good availability and performance. Federated systems like Mastodon or Matrix.org make this possible, so I really wouldn’t say that federation is always the „worst of all worlds“.

I have been using Matrix/Riot for a few months now and had about the same experience (I never used E2E though). I know that Riot still has some pain points that are quite annoying, but I love the idea behind the project and would really like to see it become more popular in the future.

FYI, they are currently working on device key cross-signing so that you don’t have to verify every single device.