Have you actually gone to remote settings and actually paired the volume control with the device you want to use?
Programme your remote for volume
If volume control doesn’t work automatically, you can manually programme your Siri Remote or Apple TV Remote:
- On your Apple TV 4K or Apple TV HD, go to Settings > Remotes and Devices > Volume Control.
- Select Learn New Device.
- Follow the onscreen steps to programme your Siri Remote or Apple TV Remote to control volume for your television or receiver.