My retail packaged iPhone had a few scratches on it and over a year later the SIM card slot failed. It was replaced with a refurb that was in better condition than the original. Pristine in fact. I wouldn't worry either way.
Just to be totally clear on this The white boxes are a mix of refurb and new. Apple takes a certain amount of each and boxes them and then sends them out to the Apple stores. So, you could get new, or you could get refurb. It's all chance.
Either way, the refurbs are just as good if not better than new. Apple replaces the casing entirely, tests the parts and replaces any ones that fail. They then run the phone through the same tests the new phones go through. And refurb comes with the same warranty.