The 3G had the same CPU, GPU, RAM and camera as the original iPhone, except they added 3G and GPS which, arguably, should have been there from the beginning.
My beef with the upgrade to the 4 from the 3Gs comes from the fact that they didn't really give the 4 a fast enough GPU to drive the retina display. The upgraded camera and flash were welcome improvements though.
3G and GPS definitely should have been there from the beginning but they weren't, so that does make it a pretty significant upgrade. I never really had any problem with the 4, I sold it off right as iOS 6 and the iPhone 5 were being released, so the last version I used on it was iOS 5 and it was damn fast.