If you do anything else on the HomePod, like listen to music or restart the ATV, after using it as an audio output for the Apple TV to use it with the ATV again you must then "reconnect" it.
In tvOS 12 and later hold down the "Home" button until you see the control center slide in from the right side of the screen. Scroll down to the AirPlay icon and select it. Then select the HomePod as the audio out for the ATV. You must do this every time. Unfortunately the ATV will not autoswitch.