Step 1
- the user activates iCloud Photos, basically surrendering his photos to Apple servers, like calling the police and saying “I am going to bring the whole content of my home to the local police station”
True. This is arguably not the perception of iCloud that Apple is going for with its privacy-focused advertising, but
caveat emptor. Might change the public discussion of OS merits in the future though. Apple is often equated with privacy and seamless integration. The reality is, pick one.
Step 3
- said fingerprints are compared by a super smart trained AI to the fingerprints in the database
- the AI is needed not to look at the content of the picture (the content is no longer part of the equation since Step 2) but to have some leeway, some wiggle room to be able to catch slightly modified (cropped, etc.) versions of the known offending picture
- the system is engineered to only match the known offending old photos from the NCMEC repository, it can’t look for new/personal children-related content
"Super smart trained AI" - I work with state of the art machine learning models, and even the best of them make the occasional dumb mistakes, because ultimately it is a dumb method still far away from human thinking.
The system is looking at the content. The
NeuralHash component (your step 2) works on
"features of the image instead of the precise values of pixels," ensuring that
"perceptually and semantically similar images" get similar fingerprints. Semantically similar, that is content matching. NeuralHash analyses the image content. If it was only about matching slight modifications, perceptual similarity would be sufficient. NeuralHash does more. Thus the fingerprint is among other things a content summary. A lot depends on the detail here, which in turn depends on the undocumented features Apple is looking for and the undocumented weights and thresholds of the system. "Two pink shapes" is more generic than "two nude humans" is more generic than "two people having sex" is more generic than "a man having sex with a boy" is more generic than "a grey-haired man..." and so on. The more detailed this goes, the closer we get to pixel perfect image comparison. We know Apple does not want that, so some level of genericness is preserved. Step 3 is comparing these image content summaries with the image content summaries from NCMEC.
Step 5
- the user uploads the photos to iCloud Photos just like HE promised to do in Step 1
- now and only now the company known as Apple Inc. is involved in any way
- at this time, Apple Inc. can do one thing and one thing only: count the positive matches security vouchers
- now 2 things can happen
1) the number of positive security vouchers is smaller than the threshold —> go to step 6a
2) the number of positive security vouchers is bigger than the threshold —> go to step 6b
True. The unspecified threshold is interesting, though. We know more than one matching picture is needed (so Apple won't do anything if they have one match, even if it is a perfect match, which is peculiar in its own right), but we do not know how many. Ten? Two?
Step 6a
- the security vouchers remain unreadable gibberish till the end of times, well after we are all dead
- not even Tim Cook, the Pope, God, Thanos with all the stones, etc. can crack their multi factor encryption, it’s like granpa Abe Simpson Hellfish unit treasure in that classic Simpsons episode, you need a set number of keys to open the vault, that’s why the “threshold” system is not a policy decision that could be changed easily by Apple Inc. but a technical safeguard that’s built-in in the system: no one could ever end up in Step 6b and Step 7 because of a single unlucky wrong match (or “false positive”)
- Apple Inc. says that a good ballpark estimate of the chance of getting enough false positives to surpass the threshold is 1 in 1 trillion per year; some people dismiss this as “yeah how do I know they’re not being too optimistic” but it should be pointed out that Apple Inc. has given 3 external experts some access to the system, and that even if that quote was wrong by tenfold (1 in 10^11 instead of 1 in 10^12) it would be still be an extremely rare event (one innocent account flagged every 117 years); moreover, the order of magnitude of said quote is perfectly plausible since we’re talking about the compound probability of multiple rare events (as an example, it would be easy to get to 1 in 10^12 as the compound probability of six 1 in 10^2 rare events)
First of all: Apple trusts its users so little that it suspects all of them of CSA, and it installs a black box into their personal property to check on them. To Apple, users are potential adversaries, who need to be checked and controlled. Information from Apple to its users must be read with this premise in mind. No claim from Apple should be taken at face value.
Your description of 6a assumes that all of this is perfectly implemented, without bugs or undocumented backdoors, and that the calculation is honest. There is no reason to make these assumptions. The trillion is hyperbole even under the most generous readings, as user accounts can differ by many orders of magnitude. External experts matter little - Apple picked them, and Apple has posited itself as our adversary. There is no basis of trust to fall back on, not any more. Apple needs to open-source this tool chain, so that we all can see what is going on in there.
Step 7 - HUMAN REVIEW
- now and only now the positive security vouchers, no longer gibberish, can be looked at by a human reviewer at Apple Inc. HQ
- the human reviewer will be able to look at a low-res version on the user’s supposedly offending photo
- if the low-res photo is something innocuous like a sunset, a bucket of sand, a cat, a goldfish, etc., (and remember: the matching is based on hashes, not content, so the content won’t necessarily be children-related, that’s not the kind of similarity the AI would catch, don’t worry about the pics of your kids, they have no more probability of being accidentally flagged than any other subject), the human reviewer will acknowledge the system made an error and discard it, no automatic calls to the cops
- if the low-res photo actually looks like actual kiddie p0rn (that gotta be the worst job on Earth and these reviewer are sometimes psychologically scarred), then Apple Inc. will disable your iCloud account and maybe report you or maybe not (depending on the follow up internal investigation)
The matching
is described as taking content into account.
Also, you left out option three - the low-res photo looks like, well, the reviewer is not sure. Is it CSA or not? Are all those people adults? Consenting adults? Might be hard to tell with the blur. Is this a picture of a barely dressed kid or a young adult? If the former, is that legal? The reviewers will have to make decisions that are not nearly as clear cut as you describe. If they decide that they cannot rule out CSA and they would rather have the experts take a look, then we get to...
Step 8 - NCMEC Review
Here all bets are off, as we do not know how this works. If the questionable pics are not variants from those in their database, then they should drop the case. The only damage is several strangers having looked at private pictures. If it is a match, off to the police. What if it is not a match, but the NCMEC reviewer thinks this might be a hitherto unknown case of CSA? Can they ask the police to investigate?
