I have personally never had an issue at that temperature or even colder. I actually wear headphones when snowboarding for multiple hours and used to use them when walking from place to place in the city for up to an hour at a time. I used fairly cheap earbuds ($20) or less as well.
But I do wear my headphones in the cold even with its snowing. What I do most of the time is close my jacket over the headphones so the button can work properly. But I haven't had any headphone failure with any iPhone.