Hopefully this will cover everything.
When using the digital output on a Mac, the signal must be passed into something which can decode the signal. AV receivers almost all have the ability to decode Dolby Digital and DTS. A TOSLink cable between a Mac and the AV receiver will allow you to listen to surround sound.
If your equipment has no decoder (which is the case with 90% of computer surround sound speaker systems), you will either need some sort of decoder, or an audio interface which does surround sound for you. I would personally go for a Creative USB SoundBlaster X-Fi. It allows you to do DTS and Dolby Digital (which is actually all handled by your video playing software).
Bear in mind though, these are all more for multimedia. If recording is your application and you want to do surround sound, this is not the route to follow.