Well it makes no difference unless you are talking about the ability to use the Remote App that can run your AppleTV.
It's not necessary, but it makes the experience better.
Having the ability to use that App is a bonus, but not having an iPhone or iPod touch is far from a deal-breaker.