it IS possible to downgrade from ios5 to is04...using tinyumbrella..
i did it with Apple tv from ios4.4.4(which is ios 5) to ios4.3 !
That's great. But I'm assuming the Apple TV you've got ISN'T an A5 device...correct? If so, then what you just said has absolutely no bearing on what we're talking about. At this time, there is NO WAY to downgrade an A5 device (iPhone 4S and iPad 2). TU will SAVE your shsh's, but they're worthless RIGHT NOW if you've got an A5 device (again...iPhone 4S and iPad 2). Again...I would LOVE to be wrong about this.
Hopefully this is something that gets figured out soon, because if not, jailbreaking isn't going to look nearly as attractive. It'll be great in the short term, but should something go wrong and you need to restore, you'll only be able to restore to the firmware that's currently being signed. If that firmware isn't jailbreakable, you're out of luck.