OK, try this.
1. Launch DVD Player app without having a DVD inserted or open.
2. Make sure your receiver is on and set to accept optical input from the channel connected to your Mac by optical cable. On my Onkyos, this has to be set for each input source. Not sure how yours is.
3. There's a setting in DVD Player that provides digital audio output and it's only available if you have an optical input (as described in 2). I'm at work, so I cannot tell you precisely where it is, but I recall it's under Preferences -> Disc Setup.
I'm pretty certain that your receiver must be on and accepting optical input when you do this with DVD Player app because your mac parses it. You only need to do this once - it will activate every time you launch DVD Player app until you disable it. You can also choose DVD Player Help from the Help menu of DVD Player app. Type 'audio' in the search box to see if that finds anything. Also, I've only done this with Dolby Digital 5.1 and not DTS 5.1, so ymmv.