HomePod Disconnection

So I just got a second HomePod and set it up as a stereo pair. Listening to music is great and I set it up as the output for my Apple TV which is also great.

The issue I have is that one it was set up as the audio for my Apple TV when I asked Siri to play some music it said it didn’t know how to do that. The only way to get it to play music was to go back into the settings and put the audio back to the TV and then I could use Siri to play music.

When I only had a single HomePod I could leave it connected to the Apple TV and use Siri to play music.

Is this a bug or is this how it’s meant to be?