Looks like you can...
I don't have an iMac here to test, but it looks entirely possible.
Here's Apple specs on iMac, under Audio, it lists both the Headphone port and the Line In as supporting Optical Audio Out/In respectively:
http://www.apple.com/imac/specs.html
Found this on YouTube:
http://www.youtube.com/watch?v=M649AesqOLU He's using iMac and Headphone output, but setup should be the same for Line In.
Here's a link to the adapter on Amazon:
http://www.amazon.com/gp/product/B0002MQGRM and about half that on Monoprice:
http://www.monoprice.com/products/product.asp?c_id=104&cp_id=10423&cs_id=1042301&p_id=2671&seq=1&format=2 Optical cables are dirt cheap there too
For the Headphone 3.5 out to your stereo, you should be able to stick with what you have now, 3.5 to RCA. You may have to adjust a few settings in SysPref|Sound, as well as Utilities|Midi Setup (for more options).
There's certainly folks in here that are Audio Engineers who can probably advise you better, but for just a couple dollars you can find out for sure. The only issue I see (IMHO) is that your iMac will be doing all the DAC (Digital-Analog Converter), all the time, and will have to be on for the setup to work. It may just be easier to get one of the $20 Optical/Stereo converters on Monoprice and go direct from TV/PS3 into stereo system and leave the iMac out of the loop.
Hope this helps.
----------
Oops, forgot the link to the Optical/Stereo converter:
http://www.monoprice.com/products/product.asp?c_id=104&cp_id=10423&cs_id=1042302&p_id=6884&seq=1&format=2
Just note that this doesn't convert 5.1 to 2 channel, you'd have to set output to Stereo on TV & PS3.