If you get a refurbished unit and everything seems to work and it looks fine, I wouldn't worry about it.
However, I've received two refurbished devices that just weren't "right".
My original iPhone 3GS had a hairline crack by the mute switch. I read online that it was a known-issue with the design, and to take it to the Apple store.
Apple took the phone, and gave me a different one.
It came out of plain, white box. This replacement iPhone 3GS had a sunken-in display. I don't know if that was an issue with the rubber gasket-thing, or something under the display pushing one side more than the other. It was obvious it was pulled apart and not put together correctly.
After a few weeks of being annoyed by the display, I took it back to Apple. They agreed that it wasn't "right", and took it.
The brought another iPhone 3GS out. Again in a plain, white box.
This iPhone 3GS had scuff marks on the bottom. Like it was dropped on concrete or something. Other than that it was in OK condition - except for the Home button. It was stiff and mushy. No "click" like the past two.
Apple said the button worked well-enough, and wouldn't get me a different one.
I should have just kept my original iPhone 3GS. It was purchased on release day. Besides working fine, it was in near-mint condition and had a nice, clicky Home button.
It was also an old-bootrom device, so could be upgraded/downgraded to any version of iOS the 3GS can run. I've since learned how to replace everything on the 3GS. Back panel, front display, logic board, battery, camera, etc. I could have completely rebuilt it. The only important part is the logic board.