many people are not capable of identifying good frequency separation - this is true. in a similar way, not everyone can visually separate reds from greens. listening to these people talk about how strawberries are hard to find in a strawberry patch while discrediting those who can actually see the difference by saying 'to each their own' is somewhat frustrating.
so no - not everyone is an audiophile, but those who are speak up and get shot down. there's a huge difference between seeing how various smartphones fit into one's lifestyle - it takes usage, consideration, and a duration to form an opinion. with audio quality, it's different. you try a pair of headphones on at the store, and you immediately know its quality, as long as you can determine the audio's source isn't faulty. this is achieved by trying on the next pair of headphones from the same source. bam. you either CAN tell it sounds like ****, or you can't. those who can't, tend to buy fashionable headphones in lieu of ones that are UNANIMOUSLY determined to be of better quality, merely because it sounds better than what they're used to.
that's pretty much it. audiophiles aren't really speaking to preference. we know some people like a lot of bass, or lower highs, etc. the argument usually goose-chases around a fixed point - that being: sound quality is different from preference. and lots of uninformed (or 'colorblind') like to mash up the two.