I have a 3G with winterboard, an icon theme (matte nano), 5 rows, 5 columns, 5 icon dock, Reminder (a status bar notifier), GRiP, a custom lock screen, weather icon with transparent theme, battery control, a magnifier theme, custom signal bars, fontswap, and SBSettings. Brightness is always at minimal unless I'm outside a lot, then it's half way. 3G is always on but I turn location and wifi off if I know I won't need them.
I always have high 40s in RAM unless I leave Safari running and with SBSettings, I can free up to 60MB RAM sometimes. The only lag I have noticed is during respring. It takes longer but who cares?
I listen to podcasts almost all day using either RSSPlayer or iFile. After listening from 8am to 11am, I usually have 90% battery left. After listening 12pm to 5pm I usually have 60+%. I couldn't be happier with performance or battery life. Some things will drain it more, like talking on the phone, web browsing, mapping or leaving the screen on, and some things like games or streaming audio/video are absolute battery killers, but that's normal.
I copy important information to my PC but I never sync, backup or restore. When I install a new firmware I erase the phone and install from scratch then copy over anything I have backed up. I think that has helped keep me free from a lot of issues.
In short, there are no downsides to me and I couldn't use the phone without the freedom that comes with jailbreaking. Accessing the file system, using MobileTerminal, writing shell scripts, and using numerous other JB apps that do things Apple won't allow all make the iPhone a powerful portable computer instead of just a cool a phone with a bunch of toys.