Just because some do not hear it does not mean no one does
For some, it is night and day.
EDIT:
I used to listen to MP3 and CDs, then bought a high def dap, and with the right can, you can definitely hear the difference.
Of course, you cannot if you are commuting in a metro with noise and all, but if you take your time to listen to your music you can hear the difference quite well.
I guess it depends on equipment first, the way you listen to your music (are you paying attention or is it background music?) and of course the music you hear and of course your age, as the older you get the lesser you hear it (trained hears can hear better than non trained hears as well).
The file as well makes a huge diff, not all FLAC (DSD/DSF) are equal
And no you do not have to spend a fortune to hear the difference.