The iPhone 4 had the a4 which was the first Apple made chip, it had twice the ram, way better camera, front camera which was insane at the time, video chat, gyroscope, worlds thinnest phone, camera flash, exterior antenna, new design, retina display and so on. I've had every iPhone other than the first one and I can say the 4 was definitely the biggest upgrade.
Correction: to you, it felt like a big upgrade.
Apple loves people who are impressed with surface veneer and paper mache. In the end, the "A4" was a rebranded iPhone 3GS chip. As another poster mentioned, the 3GS actually outperformed the 4 in many GPU intensive tests. It was old tech in a shiny new wrapper. The absolute bottom of the list as far as under the hood upgrades.