I have a JB iPhone 4 on iOS5. It feels perfectly smooth to me. I also like the notification center and being able to launch the camera from the locksreen. Then there is the matter of compatibility with apps. iMessage, Reminders, and iCloud features are also nifty although most of them can be subbed for app store alternatives.
If you don't feel as though you're missing any of these, then don't sweat it.
If you don't feel as though you're missing any of these, then don't sweat it.