Ok, sounds like you're doing the right thing, but in case not, what you want to do is attach the mini-toslink adapter to one end of your toslink cable, plug the fat end into the input on the back of your sound system marked "TV Digital In", then plug the mini end all the way into the headphone jack on your Mac. At that point the Mac should recognize it and cut the internal speakers.
If you then put on the correct audio signal on the sound system (based on the labels, I'm assuming "TV" by default, but you can probably select what input the optical audio jack is mapped to in the menus), and you should hear your Mac's sound.
Note that you can't adjust volume from the Mac when you're using the optical jack--it just sends a raw digital stream of sound data to the sound system, assuming that it will do the volume adjustment. (That's actually one way to test if the Mac is recognizing the cable--trying to adjust the volume should show an icon with a circle with a line through it.)
If it doesn't seem to be working make sure you've got the cable all the way in, and that you're listening to the right output. Note that there is sometimes a gap of about a half second between when the Mac starts outputting a sound and the optical line kicks in--this is, I assume, because the Mac (or the audio system) goes to "sleep" when there's no sound at all.