Yes, 5.1 audio can be transmitted over digital optical audio connection. However, only in compressed AC3 or DTS format. Since these two formats are the only way to get multichannel audio on a DVD, optical cables work well for multichannel audio from DVDs since the data can just be passed directly to the receiver to decode. However, Apple chose AAC to compress all of the audio they sell through iTunes, including the audio tracks for the video it sells. This can't be passed over the optical cable since receivers wouldn't know what to do with AAC. Thus, the audio would have to be converted to AC3, which would involve Apple licensing an AC3 or DTS encoder. It's not impossible for Apple to do this, but I just don't see them doing it due to the high (in my estimation) effort/demand ratio.
As for your own files that you're creating, they could in theory be passed straight through the optical port (DVD player and VLC will do this), except for my previous mention of Apple's expressed indifference to such an possibility through QuickTime. Again, the more people request this from Apple the more likely it will become a reality.
Also, the situation becomes more complex when dealing with HD-DVDs and Blu-Ray discs. Since the movie industry has deemed SPDIF ports not secure enough to transfer their content, they have to be restricted from playing full-quality audio from either format. Though this won't matter too much anyways since they support new codecs that can't be passed over SPDIF natively anyways (unless they require the SPDIF port be disabled entirely, I'm still unclear as to whether that's the case.) Which brings us back to HDMI - I was actually looking at one of the new audio codecs supported, E-AC3, when I read that HDMI 1.3 was the only connection interface that had enough bandwidth for its maximum bitrate of 6144 kbit/s (almost 10x the maximum bitrate of normal AC3.) HDMI has always specified support for both audio and video, though it's rapidly evolved in the last couple years and not too many receivers support any type of audio over HDMI at the moment, let alone the new codecs.
Anyhow, Leopard around's the corner and will likely support HD-DVD and Blu-Ray, both of which mix up multichannel audio yet again (and their DRM makes it unclear how well current optical/coax SPDIF ports will be supported.) As such, I don't really see Apple doing much about multichannel audio out for now, especially for a device such as the AppleTV.
Given the tendency of products and services that rip/copy DVDs to be sued into oblivion (especially in the US and UK), I don't see Apple doing this anytime soon. CDs were fine since no CD compliant to the Red Book spec has copy protection to remove (CDs that have copy protection aren't allowed to bear the audio CD trademark.)
TV comes out).