I have found that I can tell a big difference in quality when I change the quality setting from 20 down to what I typically use which is 16.5. I do have a 60" Pioneer Kuro that I am viewing on so maybe that makes the quality difference stand out that much more to me. One other thing that I noticed when I move from 16.5 up to around 20 is that the video becomes noticably darker. I did a test on the beginning chapters of The Fifth Element and I could really see a visible difference both in the picture detail but also the brightness of the picture (with all other factors being the same - only changing the quality number - Maybe Dynaflash can comment on why this may be).
Now obviously encoding at 16.5 and 1080 transfers created huge files - almost the same as the source. To this end I would have the same issue as a previous poster where it would take forever for a movie to start over my Wirelss N network. However, I have found that the same thing is true when trying to Airplay movies taken with the iPhone 4s (a 5 minute clip can take at least a few minutes to buffer). 1080p video is just a beast to push across a Wireless N network.
In order to compromise, and since I feel that if I really want the best experience I will just plop in the Blu-ray itself, I decided to do all my blu-ray transfers at 720p. This allows me to keep the quality setting at 16.5 and get a file size that is reasonable (typically around 6-7GB for a 2 hour movie) while still producing an image that looks good on my 60" screen. At least to my eyes, encoding at 720p with a qulaity value of 16.5 looks better than a 1080p image encoded with a qulaity value of 20 (your milage may vary). And like i said, if I really want the BEST quality, I will just put in the original Blu-ray.