I too would like to see the Apple TV be the best of the best. However, there has to be some limitations on what this tiny box can. If it was the size of the Mac mini I would surely be on board with full audio/video format support.
Tiny box? Sure. But it has an Apple A10X Fusion, with Geekbench performance (Single/Multi core) around 4000/9500 (measured on iPad Pro 2nd generation). AFAIK it's still the most powerful tiny TV box in the market. The Nvidia Shield TV doesn't even reach half of its performances. Yet, as far as I know, it direct plays most if not all audio formats.
The real reason is that Apple is bound to its boring, limited codecs. mp4 only (God forbids to support the mkv containers!!), and audio up to EAC3 (DD+), no DTS allowed, with its small restricted set of specifications and/or heavy compression (the audio). Once I have opened a bug report because a 4K HDR HEVC test video that plays just fine on my MBP is stuttering on ATV 4K. This has been the answer from the developers:
"It appears that these are hev1 files. We support hev1 within HLS but not for file playback."
So not even the supported formats are fully supported!!! The same, if I understood correctly, is what's happening with Dolby Atmos on EAC3. Apple allows it from VOD services, but not from local media players like Infuse. And let's not start about Apple turning OFF the audio passthrough, which was working just fine, before tvOS 11.3.
Sorry, but performances really have little to do with it.