I connected a 5.1 speaker system to my Mac Pro 2010 (OS 10.13.6) via the built in digital optical, but I can't select anything except 2 channels in Audio MIDI Setup. What's the deal?
Optical supports 5.1. I don’t know if it can support 7.1 however.
Most media is stereo, so if you’re playing a stereo source expect stereo output.
The real question is what is the media you’re playing and what programs are you using to decode it? Do you have a receiver?
The DVD player supports multi channel output. So can iTunes under certain circumstances. VLC will pass encoded audio via tos link.
If memory serves you should have the ability to select 8 or 6 channel audio in the audio/midi section then configure speakers to 5.1. This will then output PCM surround audio. Even with 2 channel audio selected or PCM I know that VLC will pass multi channel audio (DD/DTS) direct as is.
BELIEVE may be limited to 5.1 with a hardware decoder in place. Recall trying to setup more than that at a client's office several years ago and ran into so many issues they just went with a different solution (via AJA Kona breakout box). I believe optical worked with DVDs back in the day and assume QT/VLC/iTunes would as well.
This thread may be helpful:
https://forums.macrumors.com/threads/configure-5-1-audio-out-on-mac-pro-5-1.2033103/
https://forums.macrumors.com/threads/5-1-audio-on-mac-pro.423024/
There are some really inexpensive USB cards that provide easier mapping without dedicated hardware decoder (receiver-style amplifier) in place if you want to use monitor-style speakers for each channel.