I upgraded from a 3G to the 4 and am completely happy.
That said, the benefit depends a lot on what you use the iPhone for. The basic phone functions (voice, data, text) are somewhat improved, but not dramatically so. Where the big differences exist are in the speed of the processor/graphics, the clarity of the display, and the camera capabilities, which are massively better than the 3G. But if you don't use the camera much and only run a few simple apps, maybe those differences aren't that big a deal to you. It comes down to what you need from the phone.
If you sometimes find yourself getting frustrated with slowness on the 3G, by all means upgrade; your frustration will disappear.
BTW, unlike some of the other posters, I found the performance of the 3G with iOS 4 pretty close to the performance with 3.1.3. In fact, my wife is using the 3G now and is quite happy with it.