I use to jailbreak my devices, and if you are smart about it, you should see no speed difference. The main reason I stopped jailbreaking, was my iPhone 4. There were period where the only available jailbreak would be tethered, and I could not afford to be phoneless if something were to make my phone reboot.
If you do jailbreak DO NOT get cracked apps, the idea behind installous was to let users TRY before they buy. However 99% of people took advantage of that and just kept the apps and never paid. That being said, with the new release of iOS 5.0.1 untethered jailbreak out, I'd say go for it. There is a lot of nifty tweaks and themes you can put on your device that will make it...YOURS.
Hope this helped.