Im not sure it’s working as designed. I have two HomePods setup in a pair for my tv and never ever use them for anything else like answering calls or even asking Siri anything. They’re just speakers for my TV. I have the paired HomePods set as the Audio output in settings, but have never once heard any of the UI clicks or anything from them. And I’m finding that I always have to swipe down and select them as the option.