Because then it's nothing more than an iPhone accessory.
In theory, why is this a bad thing? Why does the Apple TV need to be more than a receiver for Airplay? Why not have the device be the same device you have on your person at all times anyway? I'm not saying the current setup is perfect, but maybe a ton of "smarts" in the device that plugs into the larger screen is... Pointless?