We’re disabling HTTP/3 for the time being, which is hopefully picked up automatically upon restart.
Restarting the browser should just help. If it doesn’t, disable HTTP3 manually.
Edit: The bug was in HTTP/3, but not in “all of HTTP/3”. We solved this on the server-end. A post-mortem will be held and I’ll make sure the outcome lands on lobste.rs
Thanks for posting it here with your official hat on and being so honest! Mistakes can happen and exactly this behavior gives me confidence in the FF crew.
Yes. This is part of the “Remote settings” service, which we can use to ship or unship features (or gradually) or recover from breakage (like here!). We mostly use it as a backend for Firefox Sync and certificate revocation lists. Technically, we could also ship a new Firefox executable and undo / change settings, but that would realistically take many more hours. Needless to says, every “large” software project has these capabilities.
BTW I’d encourage you not to disable remote settings, because it also contains certificate revocation updates and (obviously) helps with cases like this here. I understand that this is causing some concern for some people. If you’re one of those, please take a look at https://support.mozilla.org/en-US/kb/how-stop-firefox-making-automatic-connections
Edit: I’m told sync is using a different backend and some of this is inaccurate. It seems that lots of folks are linking to this thread, which is why I will leave this comment, but strike-through.
CRL updates are part of the payload that the “remote settings” service provides. So, I’m not sure what you are asking. I only know of the all-or-nothing switch.
I think driib is asking “If I control the DNS for a public wifi point, can I use an NXDOMAIN for use-application-dns.net and a spoof of aus5.mozilla.org to force an update to my own (possibly evil) version of Firefox; and if so, how do I defend against that?”. But I could be wrong.
A long standing bug in Firefox suddenly got exposed because some service updated their HTTP3 implementation. Possibly on Google or Cloudflare’s side, both of which are used by Mozilla for their infrastructure. And Firefox will check in (unless told otherwise) early on, making it possible to hit the bug very early on. Resulting it Firefox being unable to load any other page with that thread. Ouch.
Huh that’s wacky. I logged on to my computer today to start work and thought I messed up my local environment somehow. Restarted my computer 3 times, and uninstalled and reinstalled Firefox twice, then I went over to Chrome to get work done.
We’re disabling HTTP/3 for the time being, which is hopefully picked up automatically upon restart. Restarting the browser should just help.
If it doesn’t, disable HTTP3 manually.Edit: The bug was in HTTP/3, but not in “all of HTTP/3”. We solved this on the server-end. A post-mortem will be held and I’ll make sure the outcome lands on lobste.rs
Do I understand from this that Mozilla can just update my browser settings remotely without my updating‽
Update: In the end, we disabled H3 on the offending server not in any client.
Thanks for posting it here with your official hat on and being so honest! Mistakes can happen and exactly this behavior gives me confidence in the FF crew.
I promised I’d get back to this thread. The retrospective is at https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ (also discussed here at https://lobste.rs/s/m1oprf/retrospective_technical_details_on)
Yes. This is part of the “Remote settings” service, which we can use to ship or unship features (or gradually) or recover from breakage (like here!). We mostly use it as a backend for Firefox Sync and certificate revocation lists. Technically, we could also ship a new Firefox executable and undo / change settings, but that would realistically take many more hours. Needless to says, every “large” software project has these capabilities.BTW I’d encourage you not to disable remote settings, because it also contains certificate revocation updates and (obviously) helps with cases like this here. I understand that this is causing some concern for some people. If you’re one of those, please take a look at https://support.mozilla.org/en-US/kb/how-stop-firefox-making-automatic-connectionsEdit: I’m told sync is using a different backend and some of this is inaccurate. It seems that lots of folks are linking to this thread, which is why I will leave this comment, but strike-through.
How do I disable Remote Settings but keep CRL updates on?
CRL updates are part of the payload that the “remote settings” service provides. So, I’m not sure what you are asking. I only know of the all-or-nothing switch.
I think driib is asking “If I control the DNS for a public wifi point, can I use an NXDOMAIN for use-application-dns.net and a spoof of aus5.mozilla.org to force an update to my own (possibly evil) version of Firefox; and if so, how do I defend against that?”. But I could be wrong.
We sign our remote settings as well as our Firefox browser updates.
Good.
I promised I’d get back to this thread. The retrospective is at https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ (also discussed here at https://lobste.rs/s/m1oprf/retrospective_technical_details_on)
😱
A long standing bug in Firefox suddenly got exposed because some service updated their HTTP3 implementation. Possibly on Google or Cloudflare’s side, both of which are used by Mozilla for their infrastructure. And Firefox will check in (unless told otherwise) early on, making it possible to hit the bug very early on. Resulting it Firefox being unable to load any other page with that thread. Ouch.
I was wondering why Firefox was suddenly making my fans go and stop loading anything! Wow, that’s pretty messed up.
I promised I’d get back to this thread. The retrospective is at https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/ (also discussed here at https://lobste.rs/s/m1oprf/retrospective_technical_details_on)
Huh that’s wacky. I logged on to my computer today to start work and thought I messed up my local environment somehow. Restarted my computer 3 times, and uninstalled and reinstalled Firefox twice, then I went over to Chrome to get work done.
Note that this can be triggered in running & working Firefox instances, and might or might not disappear (temporarily?) upon restart.
Workaround seems to be to disable http3 in about:config for now.