I have never noticed any of that. I always get the top-of-the-line card available with my iMacs and they seem very smooth to me. Of course everyone's use cases are different.
1. You shouldn't need the top of the line card for it to be useable.
2. What cards have you used? Cause the old 5k iMacs clearly had some performance issues.