I think its all down to personal taste.
Me personally, I could never carry around 2 cell phones swapping SIMs on the fly just to use speakerphone.
But like I said, it's all personal preference.
By the way, I don't really find the speaker phone that bad? I use it every now and then, and I've never had problems.