Have you tried using an optical cable with 3.5mm Toslink mini on one end and standard Toslink on the other instead of using an adapter? I have found on my sound system that adapters sometimes just don't work. I was told by a sound system shop that the light degrades through switchboxes and adapters.
I agree with OP, that adapter should work just fine with regular TOSLINK (Optical) cable. Plug the adapter to iMac headphone jack, it should activate the red light, and all you gotta do is connect it using regular TOSLINK. Nothing wrong with that. You've done it quite right although a direct mini TOSLINK to TOSLINK cable would also work.
I suggest you set Sound Preferences on OSX? Set it to Optical digital output. Have you tried that?
That's just it; there is no 'digital out' under 'choose a device for sound output' to select wether or not the adapter is plugged in or not, ie; 'digital out' should display instead of 'headphone' when plugging it in; it does not; it reverts back to 'headphone' when I plug in the adapter.
But the red light does activated, right?
And you connect the other end of TOSLINK to audio receiver, right? What's the receiver display reports? PCM input or something?
Hmm if you have Windows installed you can try and look if your S/P-DIF works okay under BootCamp. If it's no avail maybe it's the hardware problem I'm afraid.
The red light doesn't activate. The other end is in the receiver, it says 'no signal', so that would suggest no optical coming out of the audio port. I don't have Windows installed. Do you know if there is another utility to check to see if the optical part of the port is working? I did run the OSX hardware diagnostics, said all is fine.