There seems to be some sort of software bug on the iPhone 4 in regards to speaker volume.
If you do a reboot, it should be back to normal.
The way I triggered it is by turning on vibrate, then turning it back off. This dropped my speaker volume drastically. To fix it, I did the same thing, but with headphones plugged in. After removing the headphones, the speaker volume was back to normal.
It's obviously some sort of issue with the software that causes it.