Same experience as cleung here. I use headsets that came with my old iPods all the time with my iPhone. Music plays through the headset, not the iPhone's speaker. 😕
As long as the headset plugs in OK, it should work. Like treehorn mentioned, some headsets won't physically fit into the iPhone headphone jack without an adapter.
You might try resetting your iPhone by holding down the home and power buttons until the screen goes black (don't let go when you see the "Slide to Turn Off" screen pop up).