Apple tested their system against 500 000 porn images with zero false positives. The probability they will have on the order of 30 matches seems pretty low.
There are about 1 billion iPhone users.
Apple could easily be scanning 500,000 images
a day once this rolls out.
It's also
very easy to make the false assumption that, if the chance of one of your photos being a false match is (say) 1 in 1 million, then the chance of
two of your photos matching is one in a trillion, as if they were independent events like throwing a
fair dice.
False matches are
not going to be independent: Apple's own documents explain that their "NeuralHash" system is designed to produce the same hash for
visually similar images so that it won't be fooled by cropping, scaling, changes in image quality etc. So, let's say, by sheer bad fortune one of your photos triggers a false match because some element of it matches a known CSAM image. Given that
one of your photos contains something that triggers a match, it is
highly likely that you have other photos in your collection containing the same/similar visual element: even before digital made it free, a keen amateur photographer might use most of a 36 exposure film trying to get a single shot. Or maybe the false trigger is a poster on your living room wall... So that "one in a million" chance of a single random match in a random sample can easily turn into 'more likely than not' for subsequent matches.
Now, Apple ought to know that. Unfortunately, there are too many examples of people who "ought to have known that" making this common stats error, with sometime disastrous consequences. Possibly the worst - different context, same mathematical error - being this:
en.wikipedia.org
Or, a slightly different, but more general fallacy when it comes to confusing "1 in X chance of a random match" with "1 in X chance of a false accusation" is:
en.wikipedia.org
Does this prove that Apple are evil? No, but it means that people should be very critical and ask questions, and that Apple need to be very transparent about questions like "how was the 30 matches = 1-in-a-trillion calculated?" when they wouldn't be the first people to get such a calculation wrong. Also - exactly
how is the human checking going to work, and will it go beyond rubber stamping the fact that the computer has found 30 matches (are the checkers even going to have the original CSAM images that were supposedly matched?)
Also, PSA, I really wish people would stop talking about "hashes" as if they were something fundamentally safe, secure and anonymous and somehow completely different from (e.g.) tagging faces in photos. "Hash" is a general computing term that covers a multitude of techniques and applications.
Cryptographic hashes - the sort people are most likely to have encountered in password checking, or verifying downloaded documents, or in connection with cryptocurrency - are designed so that the
slightest change in the source, even an imperceptible one, will give a different hash - and would be virtually useless because changing a single pixel in an image would change the hash and prevent detection . The "NeuralHash" system Apple is using for CSAM is a "perceptual hash" that is designed to produce the same hash for images that are "visually similar" but might have been e.g. cropped, scaled or re-compressed (and Apple's document says 'e.g.' so that's not an exhaustive list). Does that prove it is unreliable? No, but it is night-and-day different from other types of "hash" and could be just as validly described as "image recognition", having more in common with face tagging than (say) using a crypto hash to validate a file.
