all apple really needs to do is make iphone/ipad apps available on apple TV and it's game over.. you'll already have a multi-touch controller!

I think this really would scare the hell out of nintendo, sony, microsoft, and google!
Exactly. Apple is building a $1 billion data center in North Carolina, and it probably isn't just for storing people's iPhone vacation videos.
iTunes made the iPod and iPhone a success. And now iPhone and iPad are successfully leveraging more than 8 years of Apple's work on iTunes, iTunes Store, and the App Store. But so far Apple TV hasn't taken off like iPod, iPhone, and iPad. Why not? I think it's because Apple is just using the current Apple TV as an experiment.
Apple does small experiments to get experience with new technologies, then develops and productizes that technology. Back in 2006, Apple published a few games for the iPod. This is about a year before iPhone OS had even been revealed to the world. You could purchase, download, and install them through the iTunes Store.
I though it was a nice idea, but it seemed like just another marketing checkbox feature. Not a big revenue stream for Apple at all. A year later, Apple released the first iPhone, with iPhone OS software updates through iTunes. And a year after that, Apple added an iPhone app store to iTunes Store and called it the App Store.
So you might say that the original iTunes iPod games were a "hobby" for Apple. But when iPhone OS products started shipping, Apple was ready to get serious and scale that hobby up into a profit center for the company.
I suspect Apple will do the same thing with Apple TV. Almost all the pieces are in place. iTunes Store, App Store, iPhone OS, the Remote app, and deals with content providers and hardware suppliers. The things that aren't quite ready yet are high speed internet access (not yet widespread enough in the US) and Apple TV itself.
Unlike Google, who seem to be after all the TV manufacturers they can do deals with, Apple should do an exclusive deal with just one company. They've already got deep connections with LG for their LCD display panels, and guess what: LG also makes TVs. Might be a good fit. Apple could do a multi-year exclusive deal like AT&T with iPhone in the US for widescreen HDTV Apple internet televisions.
I think the final thing that Apple is waiting for is market chaos. Like the MP3 player market in 2001. Many competitors, all of whom had terrible software, some of whom had nice hardware. Apple released the iPod, then iTunes, and now they have 72% market share. Apple is probably waiting for the internet TV market to become bigger, with more profit to be had, then fractured between Google, Boxee, and whoever else is on the bleeding edge. Then Apple will step in and seize the big fat middle.