Here's my fear about an Apple TV... (first, uh... what are they going to call it?) Seriously though, it's not an unwarranted concern that Apple would distribute something with limited connectivity options priced way higher than existing options. Whenever Steve got inspired to revolutionize an industry, he usually did so by chopping it down to its essence. In the case of the TV, he'd want to control what you could connect to it. I think this would be a mistake in the case of TV's as it would render existing gear useless and I don't think many people would embrace that like they do in other markets.
Secondly, while as a consumer, I applaud the centralizing of music and video in the iTunes ecosystem (still DETEST the $1.29 price), the record and movie industries are apparently much less thrilled. They see Apple as a huge threat to their ability to freely gouge customers by limiting their distribution. Look at how they're pushing back against Netflix. One could argue that they're single-handedly destroying that company (with a little help from their CEO). Apple's really good at making hardware and software and I think the rumored Siri interface would be an amazingly fresh way to get through so much content. But the challenge Apple will have is getting Cable companies and content providers to cooperate.
Lastly, I fear an ecosystem for my TV that mirrors iOS devices. Apple has a way of getting you hooked into their ecosystem then leaving you behind unless you continue to buy more hardware. That might work with a subsidized phone, but it's not going to work with a $2,000 television so their revenue stream better be more dependent on content than hardware.
