In order to use surround speakers you can:>
1) Plug into the stereo headphone/line out jack, and let the speakers generate fake surround
2) Use an optical cable from the Mac's headphone out (with an optical/minijack adaptor or cable) to an optical in on the powered speaker or a receiver - the powered speaker or receiver must have a digital decoder built in, for this to work
3) use a Griffin Firewave on the Firewire port of the Mac, which gives you three minijacks (5.1) analog outs just like a PC soundcard does, for 5.1 speakers styestems that don't have a decoder.
In order to hear 5.1, both the source material and the software playing it back, must have 5,1 sound capability.