This is great news. I’m not being sarcastic: from almost every standpoint (privacy, open / federated Web), the more trust and goodwill Facebook burns, the better.
Can someone please explain how it is possible that one malfunctioning SDK can break the entire app?
IIUC this is due to Facebook login but still, why can’t the app continue to function regularly?
Or is it broken only for whoever actually logged in with Facebook in the first place?
This is due to Facebook’s idiosynchratic engineering practices.
At least last time this happened (https://github.com/facebook/facebook-ios-sdk/issues/1373), two months ago*, just including the SDK was enough to bring your app down before it had even initialised because Facebook ran code in the Objective-C class load method, which no sane person would override, let alone do network calls in.
That’s the idiosynchratic part, but it spells disaster when combined with Facebook’s amateurish development practices: The SDK will load objects from the FB backend, parsing them with brute force, expecting them to always be well-formed. This, combined with no working internal CI/CD system, leads to situations like this, where anyone doing a configuration error on a backend service can bring down millions of clients without them even calling the code in question.
Realized today that I’m more dismayed by the tech public’s reaction to this than the failures of engineering displayed. Specifically, a bunch of people over at the orange site believe it isn’t fully preventable when the fix is simply to see bad engineering decisions for what they are: static initialization is dangerous and has no place in a SDK deployed at FB’s scope. Full stop.
Unfortunately, there are tons of SDKs out there that does static initialization. When brought this issue up with another big-name SDK during integration, the response is in the line: we need to do some initialization, and relying on SDK users to call “init” method properly is a no-go based on what we experienced. That sounds plausible, but the solution is to make sure you have init’ed upon the first call to your library, and this can be solved through tooling (adding a custom LLVM pass to call your init method if not through your public API).
It’s just shoddy code from Facebook, and it’s a hot mess, because their SDK calls home in the initializer, IIRC, and a “bad” result that causes it to throw will blow up any app that includes the SDK, even if the app doesn’t use FB login. It’s a total catastrophe.
To add my speculative point to the good substantive answers above…
Another part of the problem is our industry has never really adopted REST as a way of architecting big public APIs. So instead of accessing them over HTTP using common patterns every API is expected to have a client library implementing the API’s custom, closed RPC-over-HTTP architecture.
I’m guessing this is why a bunch of apps on my iPhone weren’t working this morning (GroupMe, Spotify, etc.). I wasn’t logged in with FB, for what it’s worth.
Google is not better. Their Crashlytics SDK on mobile sometimes cause crashes. Same thing with AdMob SDK – when someone contacts me that the mobile app doesn’t work, often I’m just sending her an ad-free version – same git commit than previously but with ads disabled – and suddenly app is working again ;(
I’ve already disabled Crashlytics for everyone. I won’t want to disable AdMob for everyone, because I’d like the apps to generate some profit for me. Fortunately I have lots of users, so a few people with disabled ads don’t make a difference.
This is great news. I’m not being sarcastic: from almost every standpoint (privacy, open / federated Web), the more trust and goodwill Facebook burns, the better.
Can someone please explain how it is possible that one malfunctioning SDK can break the entire app? IIUC this is due to Facebook login but still, why can’t the app continue to function regularly?
Or is it broken only for whoever actually logged in with Facebook in the first place?
This is due to Facebook’s idiosynchratic engineering practices.
At least last time this happened (https://github.com/facebook/facebook-ios-sdk/issues/1373), two months ago*, just including the SDK was enough to bring your app down before it had even initialised because Facebook ran code in the Objective-C class
loadmethod, which no sane person would override, let alone do network calls in.That’s the idiosynchratic part, but it spells disaster when combined with Facebook’s amateurish development practices: The SDK will load objects from the FB backend, parsing them with brute force, expecting them to always be well-formed. This, combined with no working internal CI/CD system, leads to situations like this, where anyone doing a configuration error on a backend service can bring down millions of clients without them even calling the code in question.
“Idiosyncratic” seems like an exceedingly polite way to put it.
Realized today that I’m more dismayed by the tech public’s reaction to this than the failures of engineering displayed. Specifically, a bunch of people over at the orange site believe it isn’t fully preventable when the fix is simply to see bad engineering decisions for what they are: static initialization is dangerous and has no place in a SDK deployed at FB’s scope. Full stop.
Unfortunately, there are tons of SDKs out there that does static initialization. When brought this issue up with another big-name SDK during integration, the response is in the line: we need to do some initialization, and relying on SDK users to call “init” method properly is a no-go based on what we experienced. That sounds plausible, but the solution is to make sure you have init’ed upon the first call to your library, and this can be solved through tooling (adding a custom LLVM pass to call your init method if not through your public API).
Do you have a citation for
"no working internal CI/CD system"?They have a CircleCI instance, but if it did continuous integration and delivery, it would not be all-green.
I spit water out of my mouth as I read it.
It’s just shoddy code from Facebook, and it’s a hot mess, because their SDK calls home in the initializer, IIRC, and a “bad” result that causes it to throw will blow up any app that includes the SDK, even if the app doesn’t use FB login. It’s a total catastrophe.
To add my speculative point to the good substantive answers above…
Another part of the problem is our industry has never really adopted REST as a way of architecting big public APIs. So instead of accessing them over HTTP using common patterns every API is expected to have a client library implementing the API’s custom, closed RPC-over-HTTP architecture.
(I don’t have any solutions, I’m just lamenting.)
Which industry are you referring to? Serious question.
Software. Or that part of it that creates Web APIs.
I’m guessing this is why a bunch of apps on my iPhone weren’t working this morning (GroupMe, Spotify, etc.). I wasn’t logged in with FB, for what it’s worth.
[Comment removed by author]
Google is not better. Their Crashlytics SDK on mobile sometimes cause crashes. Same thing with AdMob SDK – when someone contacts me that the mobile app doesn’t work, often I’m just sending her an ad-free version – same git commit than previously but with ads disabled – and suddenly app is working again ;(
Why are not doing this for everyone then?
I’ve already disabled Crashlytics for everyone. I won’t want to disable AdMob for everyone, because I’d like the apps to generate some profit for me. Fortunately I have lots of users, so a few people with disabled ads don’t make a difference.