With a refurbished unit you are getting a unit that is actually better than a new one. The new ones were tested by a set of automatic machines that run the same tests on thousands of units per hour. Some errors get by which is why Apple stores carry extras that are not for sale but simply to replace units that have not worked right for customers.
A refurb unit has been fully repaired and is given more testing than is done for a new unit because it is not done on an automated assembly line set up for thousands of phones, but on a separate line for just refurbished phones. In the end what you get is a phone in all the original packaging etc. that no one, but you, can tell from a new phone, but was actually tested to a higher standard.
No one should ever worry about getting a refurb product. This holds true, not only for Apple but, for most large electronics manufacturers no matter what product they make.