Most image decoding happens in process in most cases.
iMessage performs all decoding out of process but it sounds like something is passed from iMessage to something else, that then proceeds to decode in process.
I’m honestly surprised that all image decoding happens oop on android as I recall it being super expensive to do so in webkit.
We’re talking image decoding in this thread, which you generally want in process for performance. iMessage shoves it OOP, because it now does all attachment parsing OOP (see the various “blastdoor” marketing materials :-/), but clearly downstream processes don’t necessarily do so.
A cynical part of me thinks that ImageIO should make all decoding OOP unless explicitly opted in to being in process :-/
We’re talking image decoding in this thread, which you generally want in process for performance.
Yeah, this is what I lost track of :P sorry about that.
I don’t necessarily buy that you need image decoding to be in process for performance, though? Why can’t you have the same shared memory tricks that Android’s AV stuff uses for images? If it’s fast enough for video, it seems like it should be fast enough for static images too (at least for most use cases).
Strawman design: you have an input buffer and an output buffer per client and they get shared with the decoder process. Image decode performance reduces to basically a memory copy into the input buffer (assuming you don’t do any tricks to get down to zero copies) and an IPC call, right? Then you just wait for the output buffer to be filled.
I haven’t thought too hard about this design though so maybe I’m handwaiving something away that’s actually trickier.
I don’t necessarily buy that you need image decoding to be in process for performance, though?
I’m not 100% sold on it either, but browser benchmarks disagree :-/
More seriously I think OOP image decoding does add milliseconds to the decode (zomg, entire milliseconds!) and in a browser context there can be a lot of images needing to be decoded all at once, many (most?) of which are so small as to make IPC cost outweigh actual decode time - in principle I could see OOP decoding of images on such pages get noticeably slower in the sense of “people can notice 10s of ms lag”, but that’s in a context where you also have randomized network latency in the orders of 100s of ms?
As I said, I think it would be reasonable to make all decoding be OOP by default.
Why can’t you have the same shared memory tricks that Android’s AV stuff uses for images? If it’s fast enough for video, it seems like it should be fast enough for static images too (at least for most use cases).
I think that this is largely a latency issue. As I understand it, the OOP decoding of video is entirely asynchronous - the host side is not requesting new frames, the decoders are just constantly decoding the next frame and then signaling the a repaint is required, so there is (1) no data transfer from the host to the decoder process, and (2) no latency from a “I need this decoded” request through IPC.
The other bit is video decoding is generally done with hardware assistance, so what you can (and what the apple and presumably android decoders try) is keep the entire decode + compositing of video entirely in the GPU/decoder hardware. In that model the playback is:
Throw the encoded data at the decoder API - I think the Mac/iOS APIs even allow this to just be a url so that they handle the streaming as well? Presumably android does similar
The decoder API does whatever metadata decoding is needed for the container format
The host does some configuration (decode resolution, etc) based on that and essentially allocates a GPU texture, then tells the decoder to start playback into that texture. Other BS no doubt happens if DRM is involved
The decoder starts shoveling data into the decoder hardware
Whenever a frame is decoded the decoder signals the host and the host tells the gpu to refresh the view (which is just recomposing a bunch of textures)
Obviously this is the best case scenario, but for most cases of media playback that’s what happens.
The important thing about this model though is it is works well with the decode happening in a separate process because the only thing you’re doing is compositing a single shared buffer, and that buffer is being constantly reused.
For single images that trade off isn’t there: you’re paying cpu time and memory overhead cost of every image needing a shared memory allocation, its own GPU texture to support compositing, etc.
But again this is the scenario that browsers encounter. For arbitrary apps you’re not generally dealing with those scenarios so a default OOP decode seems fine, even without any significant shared memory magic.
iOS devices feature a “Lockdown Mode” which blocks this exploit. This mode seems to primarily just prevent untrusted images from loading in various different apps; it doesn’t disable all that much functionality. It might be a good idea to turn on if you use an iPhone.
Remember Snow Leopard, the MacOS X release which focused on bugfixes and performance improvements? iOS needs the same: an entire major release focused on rewriting core components in memory-safe languages. This exploit treadmill is endless…
SL didn’t do anything specific for security, certainly no rewriting of anything in memory-safe languages.
Apple has been increasing the use of swift across the system for years now which appears to be what you’re asking for?
The problem is that “rewriting core components” is a massive task, even without concerns about performance and correctness of the rewrite (and what is “core components” that is not essentially “everything” in the OS?). It will of course become easier as more of the swift<->c++ interop improves, and partial rewrites become possible.
Apple keeps getting pwned by their image codecs, time after time. It’s been a tradition since the very first iOS jailbreak.
Yes, I am surprised the codecs have not been sandboxed on iOS like they are on Android.
Most image decoding happens in process in most cases.
iMessage performs all decoding out of process but it sounds like something is passed from iMessage to something else, that then proceeds to decode in process.
I’m honestly surprised that all image decoding happens oop on android as I recall it being super expensive to do so in webkit.
Looks like it passes around shared memory handles. Here’s the docs; they were a bit hard to search for.
Oh that looks AV related, which is OoP on iOS as well afaik.
Does this handle image codecs as well?
Oh good question, I lost track of what we were talking about. I don’t actually know. Looks from that page like maybe not though?
We’re talking image decoding in this thread, which you generally want in process for performance. iMessage shoves it OOP, because it now does all attachment parsing OOP (see the various “blastdoor” marketing materials :-/), but clearly downstream processes don’t necessarily do so.
A cynical part of me thinks that ImageIO should make all decoding OOP unless explicitly opted in to being in process :-/
Yeah, this is what I lost track of :P sorry about that.
I don’t necessarily buy that you need image decoding to be in process for performance, though? Why can’t you have the same shared memory tricks that Android’s AV stuff uses for images? If it’s fast enough for video, it seems like it should be fast enough for static images too (at least for most use cases).
Strawman design: you have an input buffer and an output buffer per client and they get shared with the decoder process. Image decode performance reduces to basically a memory copy into the input buffer (assuming you don’t do any tricks to get down to zero copies) and an IPC call, right? Then you just wait for the output buffer to be filled.
I haven’t thought too hard about this design though so maybe I’m handwaiving something away that’s actually trickier.
I’m not 100% sold on it either, but browser benchmarks disagree :-/
More seriously I think OOP image decoding does add milliseconds to the decode (zomg, entire milliseconds!) and in a browser context there can be a lot of images needing to be decoded all at once, many (most?) of which are so small as to make IPC cost outweigh actual decode time - in principle I could see OOP decoding of images on such pages get noticeably slower in the sense of “people can notice 10s of ms lag”, but that’s in a context where you also have randomized network latency in the orders of 100s of ms?
As I said, I think it would be reasonable to make all decoding be OOP by default.
I think that this is largely a latency issue. As I understand it, the OOP decoding of video is entirely asynchronous - the host side is not requesting new frames, the decoders are just constantly decoding the next frame and then signaling the a repaint is required, so there is (1) no data transfer from the host to the decoder process, and (2) no latency from a “I need this decoded” request through IPC.
The other bit is video decoding is generally done with hardware assistance, so what you can (and what the apple and presumably android decoders try) is keep the entire decode + compositing of video entirely in the GPU/decoder hardware. In that model the playback is:
Throw the encoded data at the decoder API - I think the Mac/iOS APIs even allow this to just be a url so that they handle the streaming as well? Presumably android does similar
The decoder API does whatever metadata decoding is needed for the container format
The host does some configuration (decode resolution, etc) based on that and essentially allocates a GPU texture, then tells the decoder to start playback into that texture. Other BS no doubt happens if DRM is involved
The decoder starts shoveling data into the decoder hardware
Whenever a frame is decoded the decoder signals the host and the host tells the gpu to refresh the view (which is just recomposing a bunch of textures)
Obviously this is the best case scenario, but for most cases of media playback that’s what happens.
The important thing about this model though is it is works well with the decode happening in a separate process because the only thing you’re doing is compositing a single shared buffer, and that buffer is being constantly reused.
For single images that trade off isn’t there: you’re paying cpu time and memory overhead cost of every image needing a shared memory allocation, its own GPU texture to support compositing, etc.
But again this is the scenario that browsers encounter. For arbitrary apps you’re not generally dealing with those scenarios so a default OOP decode seems fine, even without any significant shared memory magic.
iOS devices feature a “Lockdown Mode” which blocks this exploit. This mode seems to primarily just prevent untrusted images from loading in various different apps; it doesn’t disable all that much functionality. It might be a good idea to turn on if you use an iPhone.
Great work, but how do you find out if you’re affected. If I thought I might have been affected I would like to know for sure.
Great question. Using the mobile verification toolkit (mvt) of course. Runs an analysis on your iDevice backups.
Thank you!
Yeah, this is a press release. Maybe the technical details will emerge after some time when people have had time to update their devices.
Remember Snow Leopard, the MacOS X release which focused on bugfixes and performance improvements? iOS needs the same: an entire major release focused on rewriting core components in memory-safe languages. This exploit treadmill is endless…
SL didn’t do anything specific for security, certainly no rewriting of anything in memory-safe languages.
Apple has been increasing the use of swift across the system for years now which appears to be what you’re asking for?
The problem is that “rewriting core components” is a massive task, even without concerns about performance and correctness of the rewrite (and what is “core components” that is not essentially “everything” in the OS?). It will of course become easier as more of the swift<->c++ interop improves, and partial rewrites become possible.