I've honestly never listened to music through headphones on my iPhone, nor have I even used the headset. For all I know, the jack doesn't work!
The problem for me is that most in-ear buds don't fit me very well. The Apple ones fit perfectly, but alas, I usually use noise-cancelling over-the-ear headphones. For me, it's more comfortable, and I don't get paranoid about blasting my eardrums to smithereens.
In a few months, it'll probably change, though. (I'll at least keep the buds in my backpack 😛 )