Why do some people think 3GS was the biggest improvement? It added a freaking compass and basic video. And obviously it was faster.
The iPhone 4 is also faster, added HD video, front camera, LED flash, gyroscope, all new design, 40% longer battery etc
The 3Gs was a clone of the 3G that had video and a compass.
It's much more than that.
3G -> 3GS:
Improved battery life
Faster processor/more memory
Better camera, with autofocus and tap to focus
Video
Compass
Oleophobic coating
Voice Control
HSDPA implementation (allows faster download speeds)
3GS -> iPhone 4:
Improved battery life
Faster processor/more memory
Better rear camera with LED flash and backside illumination sensor
Front facing camera (for FaceTime)
HD Video at 30 fps
Gyroscope
Retina display
Extra microphone for ambient noise cancellation
802.11n support
HSUPA implementation (faster upload, in addition to HSDPA)
All new design
The 3GS isn't the small improvement over 3G that you make it out to be. In contrast, the 3G added only three new changes (3G, A-GPS, and all new design) over original iPhone.
Improvements are pretty subjective though. For someone who doesn't care for certain new features of iPhone 4 and got more out of 3GS, the 3GS is going to "feel" like a bigger improvement. Strictly speaking, iP4 added 3 more features (depending on your definition of features) over 3GS compared to 3GS over 3G.