Thanks for your detailed analysis, very informative.True. This is arguably not the perception of iCloud that Apple is going for with its privacy-focused advertising, but caveat emptor. Might change the public discussion of OS merits in the future though. Apple is often equated with privacy and seamless integration. The reality is, pick one.
"Super smart trained AI" - I work with state of the art machine learning models, and even the best of them make the occasional dumb mistakes, because ultimately it is a dumb method still far away from human thinking.
The system is looking at the content. The NeuralHash component (your step 2) works on "features of the image instead of the precise values of pixels," ensuring that "perceptually and semantically similar images" get similar fingerprints. Semantically similar, that is content matching. NeuralHash analyses the image content. If it was only about matching slight modifications, perceptual similarity would be sufficient. NeuralHash does more. Thus the fingerprint is among other things a content summary. A lot depends on the detail here, which in turn depends on the undocumented features Apple is looking for and the undocumented weights and thresholds of the system. "Two pink shapes" is more generic than "two nude humans" is more generic than "two people having sex" is more generic than "a man having sex with a boy" is more generic than "a grey-haired man..." and so on. The more detailed this goes, the closer we get to pixel perfect image comparison. We know Apple does not want that, so some level of genericness is preserved. Step 3 is comparing these image content summaries with the image content summaries from NCMEC.
True. The unspecified threshold is interesting, though. We know more than one matching picture is needed (so Apple won't do anything if they have one match, even if it is a perfect match, which is peculiar in its own right), but we do not know how many. Ten? Two?
First of all: Apple trusts its users so little that it suspects all of them of CSA, and it installs a black box into their personal property to check on them. To Apple, users are potential adversaries, who need to be checked and controlled. Information from Apple to its users must be read with this premise in mind. No claim from Apple should be taken at face value.
Your description of 6a assumes that all of this is perfectly implemented, without bugs or undocumented backdoors, and that the calculation is honest. There is no reason to make these assumptions. The trillion is hyperbole even under the most generous readings, as user accounts can differ by many orders of magnitude. External experts matter little - Apple picked them, and Apple has posited itself as our adversary. There is no basis of trust to fall back on, not any more. Apple needs to open-source this tool chain, so that we all can see what is going on in there.
The matching is described as taking content into account.
Also, you left out option three - the low-res photo looks like, well, the reviewer is not sure. Is it CSA or not? Are all those people adults? Consenting adults? Might be hard to tell with the blur. Is this a picture of a barely dressed kid or a young adult? If the former, is that legal? The reviewers will have to make decisions that are not nearly as clear cut as you describe. If they decide that they cannot rule out CSA and they would rather have the experts take a look, then we get to...
Step 8 - NCMEC Review
Here all bets are off, as we do not know how this works. If the questionable pics are not variants from those in their database, then they should drop the case. The only damage is several strangers having looked at private pictures. If it is a match, off to the police. What if it is not a match, but the NCMEC reviewer thinks this might be a hitherto unknown case of CSA? Can they ask the police to investigate?
I doubt the person you quote to will be persuaded though. He apparently doesn't care and just take everything from face value and assume that's about it.
If Apple starts to trust no one, then users have no reason to trust Apple taking care of their own business well. Frankly, I believe Apple can afford losing trust from customers, just like they could afford to damage its PR in order to force the development of CSAM scan software.
I feel Apple's "announcement" is generic enough that most customers think that's the end of it. But database maintenance, internal audit, human oversight issues etc aren't being addresses properly.