The difference is likely none. The remote will almost undoubtedly be new, not refurbed. Things like remotes are not refurbished if returned. Unless returned in original factory wrapping, it'd need to be tested, cleaned and sanitized - it's not worth the cost of parts and labor.
The Apple TV itself would be the refurbished product. It carries a full warranty, and according to Apple practice, will have been tested more thoroughly than one that came off the factory line. Overall, people's experience with Apple refurbs is excellent.