I have had mixed results connecting home Pod Mini's to my 4K TV, I usually only try it when watching the odd movie that I want a little better sound experience. This usually requires seeking the pod numerous times until the selection sticks.
Last night, my experience was incredible. I would select the Pods and before I could exit, the selection would revert to TV speakers.???? I have a Yamaha WX-030 speaker that connects first time, every time. What gives Apple?
tvOS and Homepod OS 15 will bring permanent connection and system wide sound across all apps for the apple tv 4k and the homepod minis. Just hang on there.