I'm not sure if it's supposed to cut it off, but my PMG5 (Rev A dual-2.0) has also been acting up - or following the changing whims of Apple (can't tell). For a while, the mute key didn't mute optical out. Now it does (after an OS upgrade). For a while, the Mac muted the optical out when the headphones were plugged in. Now it doesn't.
Also, now, it seems to insist on using the internal speakers as the System Preference for sound output (after each boot, until I reset it) - even though it sends audio out via the optical port anyway. And the volume control keys (the speaker keys on the keyboard) allow me to "raise" and "lower" the optical-out volume - even though it doesn't change anything (before, the keys were disabled when optical out was chosen).
The behavior has changed so much over the past year, as I went from Jaguar through the various versions of 10.3, that I have no idea what "proper" behavior is....