I have a JB 2G iPT running iOS4 with multi-tasking turned on, and I, too, have noticed a significant slow down with it. Even with simple behavior, like turning a card over in solitaire (yes, I admit I do play solitaire on my iPT) can take a second or two to respond to my touch.
Given the fact that the one app that I most wanted to be able to run in the background, the audio streaming app from my public radio station, isn't updated for iOS4, so it won't run in the background, I'm seriously considering reverting to 3.1.3.
Anyone want to offer arguments for not reverting?