Incidentally, the CSAM “scan” isn’t really a “scan”, there’s no AI processing going on. There’s a list of hashes of known CSAM material. An image with a strong hash similarity is very likely to be a copy of that same image. It doesn’t really do anything about new, unknown CSAM material, for instance, because it’s a pretty “dumb” process (instead of being AI dependent). It would be incredibly difficult (though perhaps not impossible) to extend it to political uses. Political signs at a rally would actually be very difficult to detect using this technique, assuming that people have unique signs and slogans instead of using the same sign printed off the internet. Anti-government memes would be easier to detect, yes, assuming most users of the meme share the same version of it or that there aren’t “legitimate” memes sharing a meme template with the anti-government ones. It may even be less capable than that, depending on how the hashes are formed. Are they merely the cryptographic hash of a binary file of an image? Or is is calculated from the visual image of the file? The former would be very easy to defeat, by adding garbage data and metadata to pad the file and would be useless at detecting images of political speech (unless everyone shares the exact same file). The latter means that creating new hashes is more time intensive and that it would be harder to separate “legitimate” content from political content the more similarities they have (kinda like a chameleon). You’d have to be doing some AI processing to address the chameleon exploit (where illegal speech masquerades as legal speech). And it would incidentally be far easier to add those AI processing steps surreptitiously server-side than client-side, considering that security researchers can time, perform debugging, and conduct other analytic processes far more easily by analyzing on-device functions instead of a remote server.
On-device is probably less easy to be abused for political purposes than on-server. It’s okay, but probably not objectively justifiable, to claim that the on-device analysis crosses a bridge that devices shouldn’t do, but it’s demonstrably wrong to claim that on-device is somehow more capable to violate privacy than on-server. I’m not persuaded that on-device analysis (and I’m not sure how much analysis of the content is actually going on) adds more potential to be exploited (by malicious third parties), any more so than any automatic AI scanning process. As for potential to exploit by governments or Apple, I think the increased ability for researchers to scrutinize on-device analysis provides a level of oversight that we don’t get with an on-server approach, which helps protect users. Is doing nothing less intrusive than on-device scanning? Sure, but on-device is probably far less intrusive than on-server scanning. I think people really conflated the iMessage nudity blocking feature on children’s accounts with the CSAM steps, plus, a lot of the arguments on MacRumors against them seem to be heavily based in emotional appeals vs an actual understanding of the technical aspects.
