-
Apple to scan U.S. iPhones for images of child sexual abuse apnews.com
Apple’s New ‘Child Safety’ Initiatives, and the Slippery Slope daringfireball.net
via
Vaelatern
2 years ago
Expanded Protections for Children: Frequently Asked Questions apple.com
via
kevinc
2 years ago
On a technical level it’s implemented very well.
It is matching against a list, so unlike a general recognition AI, there’s very little chance of misidentification.
The blocklist and matching process is split between client-side and server-side, so it can’t be easily extracted from the phone for nefarious purposes.
Apple has spent a considerable effort to cryptographically ensure they know nothing until multiple matches are found. Phone even sends dummy traffic to obscure how many potential matches are there.
So as far as scanning for the intended purpose, it’s a careful well thought-out design.
I am worried about governments putting pressure on Apple to add more kinds of unwanted images to this list. The list is opaque, and for obvious reasons, it can’t be reviewed.
This is an improvement over their existing policy of giving authoritarian governments access to iCloud keys for their users: https://www.reuters.com/article/us-china-apple-icloud-insight/apple-moves-to-store-icloud-keys-in-china-raising-human-rights-fears-idUSKCN1G8060
This technology will allow Apple to expose only content that governments specifically ban rather than having to give them access to everything. We should be celebrating this for both its ability to combat child abuse and that it protects Apple’s customers from over-broad privacy invasion.
Do governments always make fair and righteous decisions in when deciding what images to ban? I see this situation as disastrous for human rights because you know darn well countries like China will bully Apple into including whatever images they want in that database.
But China including whatever images they want is WAY better for privacy than today when China simply has access to all of Apple’s Chinese users’ data.
That’s not the case, unless you mean to say China bullying Apple into giving them a user’s decryption key? That scenario is possible with or without this system.
This has been the status-quo for the past 3.5 years: https://www.reuters.com/article/us-china-apple-icloud-insight/apple-moves-to-store-icloud-keys-in-china-raising-human-rights-fears-idUSKCN1G8060
China demand access to user data so many large American tech companies don’t have a significant presence there. Some American companies that are less committed to privacy comply with the conditions that China places for operating there. It’s a huge market so it’s been a great business move for Apple.
Having the ability to scan users’ content in device might be a way to achieve censorship without such indiscriminate access to user data.
The article makes many speculations, but there is nothing concrete regarding the Chinese government having the kind of access you described written in it.
Also see this more recent article: https://www.nytimes.com/2021/05/17/technology/apple-china-censorship-data.html
Apple user data in China is not controlled by Apple, it’s controlled by GCBD, a company owned by a Chinese regional government. Instead of using standard HSMs they use a hacked up iOS system. Apple’s security chips are vulnerable to local attacks. https://arstechnica.com/information-technology/2020/10/apples-t2-security-chip-has-an-unfixable-flaw/
So there’s a government owned company that controls the user data which is encrypted with keys stored in an insecure system. If user data is not being accessed that’s a choice that the Chinese government is making, not a restriction on their access.
GCBD is the Chinese company that provides apple with datacenter type services. This is not the same as “controls the user data”.
From the New York Times article you linked:
So to get around US privacy laws and comply with Chinese surveillance laws a Chinese government owned company is the iCloud “service provider” (with Apple listed as an “additional party”) and per the ToS “will have access to all data that you store on this service”.
It was a great business decision. They’re the only major western tech company making a lot of money from the huge Chinese market. I personally wouldn’t want to work there but the people who do are doing very well.
Could such a feature be “pretty easily” fooled to trigger law enforcement to someone as the article implies?
Is it plausible to assume that they scan the cached Telegram/Whatsapp/Browser images? If so, how would it behave if someone sends you a set of known infractor images? (an evil chat bot, for example)
Apple says they scan only images in the iCloud library, so images in 3rd party apps and browsers won’t be scanned, unless you save them or screenshot them to your iCloud library. Of course, Apple devices belong to Apple, not you, so Apple could later decide to scan whatever they want.
With the current scheme, to cause someone trouble, you’d first have to have multiple banned images to send to them. I hope obtaining actual CSAM is not “pretty easy”.
My big worry was that a plaintext blocklist on the phone could be used to generate arbitrary new matching images, but fortunately Apple’s scheme protects against this — the phone doesn’t know if images match. Therefore, you can’t easily make innocent-looking images to trick someone to save them.
Is there a source for this information?
What’s your source for the “multiple banned images” part? Skimmed through Apple’s technical PDF descriptions a bit but didn’t find that part right away.
Threshold Secret Sharing of Safety Vouchers
https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf
I believe pictures in a lot of messaging apps are automatically uploaded to iCloud. So you could just send someone some pictures over WhatsApp, email, or whatnot. Not 100% sure of this though; I’d have to check. I disabled all the iCloud stuff because it kept nagging.
That or you can generate adversarial images that trigger known hashes. It isn’t using cryptographic hashes, it is using perceptual hashes.
No, you can’t, because the device doesn’t know if it has got a match.
And you think there will be no other way to get ahold of any of the perceptual hashes that are being scanned for?
What I’m saying is that you can’t easily abuse Apple’s implementation for this. They’ve anticipated that problem and defended against it.
If you get hold of some hashes or banned images from another source, that’s not Apple’s fault.
I think the greatest concern I have is the potential for governments to pressure Apple into using the tech for other purposes since we know Apple loves them some iPhone sales. That being said though, their technical summary seems to rule out every other concern since it only scans images before uploading them, only flags matches against a known set of images, and those matches are manually reviewed. You couldn’t really fake a match unless you knew the existing dataset so it’s almost impossible that you could “SWAT” someone, and even then it would be trivial to demonstrate that the image was sent to you by someone else. Perhaps if someone had your credentials they could upload images to your account but then that was a risk long before this.
I don’t like the idea of this being used for other types of images, but as implemented and for the purpose given it seems like a pretty well thought-out system. I am totally fine with the pushback since it makes Apple be as transparent as possible, but I don’t like that people are making some false claims about how the tech works. I think the focus of criticism deserves to be squarely on the issue of whether Apple bends to pressure from more restrictive governments when their profits are on the line.
I hope they are prepared for the difficulty on the people reviewing. I recall reports of police officers doing this sort of work having mental health issues and not doing it for very long.
Agreed. I have heard that the Facebook team that handles these sorts of things has extremely high turnover.
It looks like the manual review process involves low-resolution versions of the image to protect reviewers.
I’d be far more worried if they didn’t have high turnover.
[Comment removed by author]
This was also true of DVD private keys until it wasn’t. (This is not to negate the second part of your sentence, only the first.)
Please consider signing the open letter against these changes: https://appleprivacyletter.com/
Are you going to post an open letter for Microsoft, Google, DropBox, Facebook, Twitter, and all the other companies who have used the exact same database for this exact purpose for the last decade?
Which provider has previously used this list against images that aren’t stored on their infrastructure?
Images sent via iMessage are stored on Apple’s infrastructure.
I think the question had implied “stored in plain text”. iMessage doesn’t do that.
Right. So, every other provider has direct access to your photos, and scans for CSAM with their direct access. Apple, rather than give up their E2E messaging, has devised a privacy-preserving scheme to perform these scans directly on client devices.
I really don’t understand how Apple is the bad guy here.
Other providers that scan cleartext images are off the hook, because they’ve never had E2E privacy guarantee.
[smart guy meme]: You can’t have encryption backdoor if you don’t have encryption.
Apple’s E2E used to be a strong guarantee, but this scanning is a hole in it. Countries that have secret courts, gag orders, and national security letters can easily demand that Apple slip in a few more hashes. It’s not possible for anyone else to verify what these hashes actually match and where they came from. This is effectively an encryption backdoor.
If I understood what I read, although the private set intersection is done on device, it’s only done for photos that are synced with iCloud Photo Library.
Apologies to all in this thread. Like many I originally misunderstood what Apple was doing. This post was based on that misunderstanding, and now I’m not sure what to do about it. Disowning feels like the opposite of acknowledging my mistake, but now I have 8 voted based on being a dumbass 🙁
iCloud Photos are stored on Apple infrastructure.
This page gets the scope of scanning wrong in the second paragraph, so I’m not sure it’s well researched.
how so? can you explain?
“Apple’s proposed technology works by continuously monitoring all photos stored or shared on a user’s iPhone, iPad or Mac, and notifying the authorities if a certain number of objectionable photos is detected.”
seems like an appropriate high-level description of what is being done, how is it wrong?
I may be wrong but, from what I understood, a team of reviewers is notified to check manually the photos once a certain number of objectionable photos is detected, not the authorities… If (and only if) the team of reviewers agrees with the hashes matches, they notify the authorities.
This is a detail but this introduces a manual verification before notifying the authorities, which is important.
From MacRumors:
Link to the resource: https://www.macrumors.com/2021/08/05/apple-csam-detection-disabled-icloud-photos/
Second paragraph of the AP article
This resource from Apple also states that only images uploaded to iCloud are scanned.
This quote you cite figures nowhere within the page.
[Comment removed by author]
You replied to my comment linking to an open letter, you didn’t post a top-level comment.
Only photos uploaded to iCloud Photos are matched against known hashes.
Or just don’t buy an Apple device. Do you really think a trillion dollar company cares about digital signatures?
I think this is a good statement of intent though.
I just bought an iPhone 12 and would be otherwise unlikely to be noticed as a lost sale until the iPhone 14~ since most people don’t upgrade a single minor version.
Giving them warning that they have lost me as a customer because of this is a good signal for them. If they choose not to listen then that’s fine, they made a choice.
Also the more noise we make as a community; the more this topic gains attention from those not in the industry.
I didn’t mean to make some sort of “statement” to Apple. I find that idea laughable. What I meant is that if you are really concerned about your privacy to the point where scanning for illegal images is “threaten[ing] to undermine fundamental privacy protections” (which I think is reasonable), then why buy Apple in the first place? This isn’t the first time they have violated their users’ privacy, and it certainly wont be the last.
What’s your proposed alternative?
I think Apple making a stance on privacy, often posturing about it a lot, does cause a lot of good will and generally those who prefer to maintain privacy have been buying their products. (myself included). You can argue that it’s folly but the alternatives are akin to growing your own vegetables on a plot of land in the middle of nowhere connected to no grid (a-la rooted android phones with f-droid) or google owned devices which have a significantly worse privacy track record.
You oughta update your intel about the “alternative” smartphone space. Things have come a long way from “growing your own vegetables on a plot of land in the middle of nowhere connected to no grid.” The big two user-friendly options are CalyxOS and LineageOS with microG. If you don’t feel like installing an OS yourself, the Calyx Institute, the 501(c)(3) nonprofit which develops CalyxOS, even offers the Pixel 4a with CalyxOS preinstalled for about $600.
I’m running LineageOS on a OnePlus 6T, and everything works, even banking apps. The experience is somewhere between “nearly identical” and “somewhat improved” relative to that of the operating system which came with the phone. I think the local optimum between privacy-friendliness and user-friendliness in the smartphone world is more obvious than ever, and iOS sure ain’t it these days.
It does seem folly to make a statement by not buying something, but consider this: When you vote, there are myriad ways that politicians have to dilute your impact (not going to enumerate them here but it’s easy to do). By comparison, when you make an economic choice, ever dollar is counted in full, one way or another. So if you vote, and you should, then there’s every reason to vote with your pocketbook as well.
Sounds reasonable at first, but it’s only the tip of the wedge. Or rather, another broadside in the ongoing power struggle between computers owners and service providers and in the fight against general purpose computing.
If the government did this it would be a 4th Amendment violation.
That’s probably why this technology—and this database of known child pornography—hasn’t been abused by the government during the last decade it has been in use by every other major tech company.
Full stop. color me surprised… shoulda known better, but I wanted to believe.
This quote is a little misleading due to being vague. “Apple plans to scan users’ encrypted messages” sounds like decrypting and reading all your messages on their servers where Apple employees can see. That’s not how it works. It’s not even part of the iMessage service; it’s in the Messages app itself.
Thank you, I think the news is overblown. I have to remind myself that most news stories are largely embellished, if not totally devoid of factual evidence. Nevertheless, I am looking for privacy friendly alternatives, it’s a shame the fairphone isn’t sold directly in the US…
Is it me, or does a dataset of 200.000 pictures seem a bit small?
For now. Who’s to say authorities won’t ask to scan photos for known terrorists, criminals, or political agitators? Or how long until Apple is “forced” to scan phones directly because pedophiles are avoiding the Apple Cloud?
That’s not how the technology works. It matches known images only. Like PhotoDNA—the original technology used for this purpose—it’s resistant to things like cropping, resizing, or re-encoding. But it can’t do things like facial recognition, it only detects a fixed set of images compiled by various authorities and tech companies. Read this technical summary from Apple.
FWIW, most major tech companies that host images have been matching against this shared database for years. Google Photos, Drive, Gmail, DropBox, OneDrive, and plenty more things commonly used on both iPhones and Androids. Apple is a decade late to this party—I’m genuinely surprised they haven’t been doing this already.
Apple does scan this when it hits iCloud.
The difference is now they’re making your phone scan it’s own photos before they ever leave your device.
Only if they are uploaded to iCloud. I understand it feels iffy that the matching against known bad hashes is done on-device, but this could be a way to implement E2E for iCloud Photos later on.
But one match is enough. The goal is to detect, not to rate.
This by far the scariest thing Apple will ever do. And of course it is al under the guise of “child safety” how can anyone dispute that?
This is making me sick. To think that any arbitrary image or message you send or recieve will be scanned by Apple… I just don’t understand how they can run ads claiming to be private while scanning every iPhone user’s content.
I will never own an iPhone, fuck Apple.
That’s not what is happening here. Only children whose parents opt them into the Message app’s new explicit content filter have Message scanning taking place and it is taking place locally. No communication with Apple. For adult users the only scanning taking place is hash matching if you upload a picture to iCloud against a database of known illegal content.
Please tell me you aren’t defending this.
“Known illegal content” mean CSAM right now, but will eventually become anything the government or big tech finds to be undesirable. You can’t be that naive, can you?
This sets the stage for full blown device scans in the future and it cannot stand.
Please see my comment from yesterday for an overview of my position.
I think it’s worth asking what you mean when you say “scanned by Apple.” It sounds like you mean scanned in the cloud on Apple servers where they can read it, but that’s not the case here.
At first I thought this would be using a known database of CSE material, stored in hashed form or such, that could simply be indexed against and thought it was a great idea. To me, that is the ideal non-intrusive method of identifying users storing illegal material of this nature (and, best case scenario, finding a collision within some hashing method). Finding out that it’s an “AI” that scans all your images and sends them to an Apple-approved review board was really disheartening. It feels like this is just a convoluted method for Apple to get governments and grant agencies to fund their AI research in some way.
Did you read the technical summary from Apple? Sounds like a modern take on PhotoDNA using a neural network to me.
Mostly ignorant questions: Do we not know from adversarial classifier research that, given “similar” and “different” are meant to be meaningful in a human sense, this is impossible? Are they going to publish their false positive rate? Has this been tested against adversarial images or just with a normal dataset?
I’m not 100% sure what you’re asking, but NeuralHash isn’t a classifier in the usual sense, i.e. it can’t say “this is a picture of a dog” or whatever. It has specifically been trained to detect if two images are literally the same image.
Consider trying to detect known bad images with SHA-256. Someone wanting to circumvent that detection could save as png instead of jpg, or crop 1px off the side, or brighten by 1%. Any small change like that would defeat fingerprinting based on SHA-256 or other conventional hashing algorithms. Clearly this problem calls for another solution.
NeuralHash—and its predecessor PhotoDNA, a procedural algorithm—doesn’t make any inferences about the conceptual contents of the image. Instead it generates a hash that can match two images together even if they’ve been edited. The images don’t have to be similar or different in a strictly human sense. For example, NeuralHash would not produce a match for two different images of paint splatter. The edges of the splatters would be at different distances and angles relative to each other. Humans may consider two such images as visually similar, or perhaps even indistinguishable without close inspection, but NeuralHash doesn’t measure that sort of visual similarity.
If it’s not literally a SHA of the file, then it’s some kind of imprecise comparison, and calling it “neural” sure makes it sound like a classifier. Presumably the result of the comparison algorithm is supposed to be “a human would consider B to be an edited version of A”. Otherwise it would again be trivial to fool, just by changing one pixel.
So if it’s an imprecise algorithm supposedly mimicking a human decision, the natural question is, what happens if an adversary tries to induce a false positive? Impossible with a file hash, but I’m suspecting significantly less impossible with this algorithm. At any rate, I’m just asking, has anyone tried?
Using the name “Neural” likely just means the hashing algorithm has been optimised to run on the Apple Neural Engine accelerator, for the sake of speed and power efficiency.
What happens if an adversary tries to induce a false positive? About the only consequence I can think of is that the person at Apple who performs the human review step has a slightly nicer day because they got to look at an image which isn’t CSAM.
According to the article:
So it would seem so.
I’m not entirely sure if this needs to be an issue as such. If a match triggers an immediate automatic shutdown of your account: then yes, it’s a problem. But as I read this, it merely “flags” your account for closer inspection, and you may not even know it happened. And as I understand it, false matches are mostly limited to intentionally crafted images, rather than accidental false match (like e.g. Tumblr’s mess of a “nudity detector”).
The bigger issue is sending of actual child pornography: you don’t control the messages you receive over email, iMessage, WhatsApp, etc. and a malicious “here are the pictures you asked for 😉” message could potentially land you in some problems. It essentially opens you up to a new type of “digital swatting” attack and also one that could potentially be hard to defend from in cases of persistent and motivated attackers.
Photos over Messages are not automatically added to iCloud Photos. WhatsApp has an option to do that, I think.
One of the concerns that occured to me is that if adversarial testing hasn’t been a focus, it might turn out to be pretty easy to generate a false positive, and then you’ve not only got a 4chan-ready DoS attack on iCloud but a “mass swatting” opportunity. Imagine an adversarial meme image goes popular, and once you receive ten of them your account gets locked and you’re reported to the authorities.
But some meme is harmless? Who cares if some harmless meme is reported to the authorities.
[Comment removed by author]
You might be confusing the Messages feature and the iCloud feature.
I foresee lots of horny teenagers causing lots of very awkward problems.
How so? Do teenagers frequently exchange known images of child pornography that the relevant authorities have already reviewed for inclusion in this database?
To be fair, the Messages feature doesn’t match against a list of hashes but actually does do ML to detect suspicious photos. That said, it’s only for under-13s, who are not teenagers.
Additionally, that particular feature does not communicate with Apple at all. Only the parents or guardians on the account. So maybe an awkward conversation with a parent but Apple never gets involved.
How will this impact battery lifetime? Can an estimation be given by what has been put forward by Apple?
Relative to the ML processing done to classify the photo in other ways (e.g. contains a dog) it would be a minuscule drop in the ocean. Processing of the back catalogue would almost certainly be restricted to when the phone is locked and charging.
There goes all credible neutrality…
Merge into
ihvn95, please.I do appreciate the counterpoint here (well, less counterpoint and more “actually reading the thing carefully”), though of course, people will probably dismiss Gruber on the topic.
Another merge into
ihvn95.(Again, I get the impression everyone talking about this is conflating multiple things and no one seems to be able to exactly describe it. See the Gruber post merged into the OP.)
Agreed. This FAQ even begins by trying to disambiguate the two features.
I’m confused. They are doing all of this fancy cryptography for photos uploaded to iCloud but according to https://support.apple.com/en-us/HT202303 photos in iCloud are not end-to-end encrypted. So what is the point of all this cryptography?
More correctly, apple to scan iPhones for a set of images provided by a government and then report any user with any of those images to said government.
This is clearly not a technology that is going to be abused.
More correctly, Apple to hash photos prior to being sent to iCloud; compares this has to a set provided by a non-Government non-profit (NCMEC). Any user with multiple hash matches has these matched images human reviewed by an Apple employee; if they are confirmed to be CSAM, their iCloud account is closed and a report is filed with a non-Government non-profit (NCMEC).
NCMEC was created by the US congress, has special legal exemptions that apply only to it. It is a government agency in all but name.
But that’s actually moot, because now that apple has created this “feature” all a government has to do is pass a law requiring apple to search for whatever images they want and apple has no choice.
Hence, this is a technology that allows a government to decide what you can have on your device, and identifies you to that government if you don’t comply.
The Government could have passed a law requiring Google to search for whatever images they want and Google would have no choice. So why haven’t they? Apple’s the one late to the party, and now everyone’s piling on Apple for the existence of the party. It’s hypocrisy.
I’m curious, can you link to where you publicly complained when Google (and Facebook, Microsoft, etc etc) implemented the same thing? Or are you only angry because it’s fashionable to criticise Apple?
The confusion and misunderstanding around this issue is far out. Apple spends so much effort designing software to be approachable and understood, but their communication around these features has been dismissed by many people and not understood. I have to imagine there’s a better way to introduce these features.