I would have thought by now there would have been AI routines that could be used in the OS to help block this sort of thing. Also, it doesn't help that programmers have a tradition of naming files in an inscrutable way. Some sort of naming convention should be required, so that picking up malware is easier.
Um. I may be naïve in responding to this, but I have poor impulse control. Caveat Asperger's, it's a brain dump it'd take me hours to condense…
"Artificial Intelligence" is not an incantation. It is mathematically impossible for a program to do that kind of work perfectly. It may do well against a random challenge, but once the weakness is found (as in time it will be, to a motivated attacker), they are specifically aimed at the part that fails.
Also, not every user is perfectly careful, and not every user can live with the safeguards at full force. Developers often have to relax them: macOS comes with rigid guards against potentially-risky behavior that some benign software has to perform. Threre are several layers; developers may turn some off on their own machines; users may waive protection for a particular app or across the board. (Until recently Kerbal Space Program was a very tempting app that was not signed.)
The latter gets you into the realm of "social engineering:" The malware author plays on the user's impulses (I have to see that movie/play that game/get rich quick/spy on my partner/you-know-the-rest
NOW). And the user opens the door. The exploit doesn't have to be foolproof, technically or personally. The attacker doesn't care about
you, he cares about lots of
somebodys. The defense
does have to be foolproof.
A few stray points and corollaries:
The ship sailed on forbidding "untrusted" software across-the-board in the mid-1980s. It's not in Apple's interest to make it harder for developers to hand out software to a small circle of friends and clients — it's a minority platform, Apple isn't in Microsoft's position (or culture) of making demands. Apple was burned badly by its legal team's proposal in 1987(?) after-the-fact to license the right to create Mac software. The openings were baked-in from the start, and Apple would rather not absolutely break existing products.
So why not make the trusted developer program free? Apple loses money on the $99 registration fee, I guarantee it. The main purpose is to prove you have enough of a relationship with a bank to transfer a small but nontrivial amount of money. Once that happens Apple "knows where you live." (It's no more perfect than anything else, but it filters out most of the hax0r d00dz who aren't the masterminds behind dozens of banking cutouts.)
Just as AI would have false negatives it would have false positives. The only alternative is to catch some cases and counter them by human intervention. That's what Apple does by revoking trust for apps that are signed. It has macOS poll Apple for fresh malware rules every few days. It issues security updates every few months that outright ban earlier versions of Flash. (Okay, not just that, but Flash is usually a marquee feature of the update.)
If you want an "app store only" policy, it's available in the Security panel.
If by cryptic file names you mean things like "com.example.exampleapp.syncd.plist," they're unavoidable. Developers can't risk using the same names as others. Some names are imposed by the OS to group related files — they are coordinated to the extent possible, just not so it's obvious to amateurs. "syncd" is not cryptic to people who know enough to monkey with that kind of file; the professional jargon says that this is for a background task that does some kind of synchronization for Example App Pro. Or so it says. Many, many apps have to create multiple files which have to have unique names; the best way to do so is to use a UUID (large-ish partly-random but nonsensical number) for the name.
Blindly removing such files may work, or work well enough, or not work at all, or do harm, depending on whether they depend upon, or are depended upon by, system registries or other services that you do not want to kill or let eat into performance and behavior while good software thrashes to find them. Meanwhile the malware vestiges you left (did they graft themselves into your video driver?) re-create the files, possibly under different names in hopes of finding ones you won't suspect.
Besides, it does occur to hacker masterminds to use names like com.microsoft.office.common instead of ru.legion.of.evil.passwordstealer.