The lossy compression just destroys too much of the original information.
From earlier in the thread, Apple has posted some audio tools for downloade. One is a plugin that allows doing comparisons between source material and how it sounds after their lossy encoding. It includes an ABX test, I'd bet you don't hear the difference nearly as well as you think you do.
http://www.apple.com/itunes/mastered-for-itunes/
I personally like to use 88.2 because it's an even multiple of the CD spec so you don't run into dither issues.
A common misconception, with a sample rate converter that's designed properly it upsamples so there's no difference between converting from 96 or from 88.2. The "even multiple" thing makes no difference at all.