Like others have posted, the jailbreak itself in my experience does not slow performance. What you install may impact performance.
I don't install a lot of tweaks but of what I have installed I have found Winterboard to cause the biggest hit to performance. Other tweaks, like 3GUnrestrictor, InfiniFolders, NoNewsIsGoodNews don't in my experience slow the phone down. Things like apps (PDA Net, MXTube) shouldn't impact performance at all as they aren't tweaks running all the time, they're apps that only consume resources when you're using them, (like most apps in the app store).
It also really depends on your device as well. I have an iPhone 4S and iPad 2 and they can handle what you throw at them pretty well. An iPhone 3GS, iPhone 4, and iPad (1) will struggle more with the tweaks than the newer models (obviously).
I recommend doing the jailbreak and slowly adding things over time and watch the performance. If you add something that causes a performance hit, drains the battery, or makes the phone unstable, you can usually just remove it and the problem goes away.